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EXECUTIVE  SUMMARY 


This  research  offers  three  principal  families  of  recommendations  to  improve  assessments 
in  the  Global  Peace  Operations  Initiative  (GPOI).  The  first  recommendation  is  simply  to 
base  measurements  on  stated  objectives  and  end  states,  which  is  an  expression  of  the 
basic  systems  engineering  concept  of  traceability.  Secondly,  this  research  advocates  that 
the  stakeholders  think  in  a  more  holistic  manner  that  should  be  exhibited  both  by 
broadening  the  solution  space  beyond  technical  means  to  include  organizational 
dynamics,  as  well  as  understanding  that  to  assess  any  given  element  of  a  system  likely 
requires  the  measurement  of  its  connections  with  other  system  elements.  Thirdly,  given 
the  complexity  and  ill-defined  nature  of  the  problem,  this  research  advises  the  application 
of  systems  architecting  to  provide  a  structured  framework  from  which  measurement 
models  can  then  be  derived. 

When  viewed  as  a  system,  the  problem  facing  the  GPOI  is  a  deficient  feedback 
mechanism.  This  lack  of  feedback  makes  difficult  both  the  day  to  day  operations  of 
guiding  investments  as  well  as  the  task  of  explaining  and  justifying  the  program’s  impact 
to  resource  sponsors.  This  research  has  concluded  that  the  GPOI  Assessments  framework 
operates  primarily  at  the  verification  level.  Verification  answers  the  question  of  the  sort: 
did  we  build  it  like  we  designed  it?  Or  did  we  do  what  we  intended  to?  In  the  case  of 
GPOI  it  would  take  the  form  of:  did  our  activities  result  in  the  construction  of  a  training 
facility  or  the  conduct  of  a  “train  the  trainer”  event  as  planned?  Verification  is  necessary, 
especially  for  any  government  agency  naturally  concerned  with  proper  accounting  and 
disbursement  of  funds.  However,  verification  does  not  answer  the  more  ultimate  “so 
what?”  questions.  Hence,  the  need  to  evolve  the  GPOI  Assessments  to  a  framework  that 
addresses  the  validation  level  question  of  “did  it  matter?”  To  be  clear,  a  “body  count”  of 
troops  who  were  trained  or  deployed,  while  a  good  metric  to  measure  for  other  reasons,  is 
not  a  validation  level  question.  A  validation  level  assessment  would  be  “how  did  those 
trained  troops  contribute  and  perform  on  UN  Peacekeeping  missions?”  or  “how  has 
regional  security  been  improved  as  a  result  of  this  expanded  capacity?” 


This  research  suggests  that  the  GPOI  objectives  and  outcomes  are  not  fully 
articulated  to  the  degree  necessary  to  build  a  coherent  assessments  framework.  The  GPOI 
Implementation  Guide  (GIG)  begins  by  alluding  to  what  the  program  intended  to 
accomplish  before  going  into  much  greater  detail  into  the  objectives  (Bureau  of  Political- 
Military  Affairs  and  Office  of  Plans,  Policy  and  Analysis  2013).  The  stated  objectives 
have  been,  in  part,  confused  with  program  outcomes  and  hence  been  made  the  focus  of 
the  assessments  framework.  The  vagueness  of  outcomes  is  in  part  to  blame  for  the  lack  of 
emphasis  on  validation  in  the  current  assessments  framework. 

First  defined  outcomes  should  be  articulated  and  then  the  supporting,  or 
actualizing,  objectives  shall  be  expressed.  From  the  preponderance  of  literature  on  the 
subject  it  would  appear  that  the  GPOI,  at  the  highest  level,  is  trying  to  achieve  two 
fundamental  outcomes:  enhanced  regional  security  for  its  partner  nations  and  an 
increased  pool  of  well-trained  peacekeepers  that  can  successful  execute  international 
peacekeeping  missions.  Figure  1  presents  these  notional  outcomes  and  illustrates  how  the 
stated  objectives  support  them. 

These  stated  objectives  and  outcomes  become  the  basis  of  measures  of 
performance  (MOP)  and  measures  of  effectiveness  (MOE),  respectively.  The  GPOI  does 
have  a  mechanism  known  as  the  Full  Training  Capability  (FTC)  Assessment,  which  maps 
directly,  and  very  thoroughly,  to  the  objective  of  “Assist  partner  countries  to  achieve  and 
sustain  FTC  in  peace  operations  training.”  What  is  needed,  of  course,  are  mechanisms  to 
assess  the  other  five  objectives  as  well  as  the  all-important  outcomes. 


Measures  of  Performance  Measures  of  Effectiveness 

(from  stated  GPOl  Objectives)  (derived  from  stated  GPOl  Mission) 


Figure  1.  Notional  Objectives  to  Outcome  Mapping  (after  Bureau  of  Political- 
Military  Affairs  and  Office  of  Plans,  Policy  and  Analysis  2013,  2-4) 


As  mentioned  earlier  the  second  family  of  recommendations  can  be  thought  of  as 
a  call  to  embrace  a  more  holistic  line  of  thinking.  This  research  suggests  that  the  GPOl 
assessments  team  widen  their  collective  aperture  in  two  ways:  the  first  is  to  consider  the 
appropriate  measurement  points  in  the  system  and  the  second  is  to  think  about  actualizing 
solutions  both  as  the  necessary  technical  elements  as  well  as  the  enabling  non-technical 
elements. 

Well  developed  (and  traceable)  MOPs  and  MOEs  are  necessary  to  assess 
achievement  of  objectives  and  outcomes.  However,  the  stakeholder  set  is  not  likely  to  be 
satisfied  with  an  understanding  of  objective  and  outcomes  by  themselves;  rather  the 
broader  question  is  to  understand  how  system  direction  impacted  system  response.  As  it 
pertains  to  GPOl,  this  would  take  the  form  of  understanding  how  the  arc  of  investment 
(such  as  money,  organizational  resources)  enables  GPOl  activities  (e.g.,  building  a  school 
house,  providing  deploying  units  with  equipage)  that  achieve  objectives  (e.g.,  supporting 
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deploying  units)  that  in  turn  realize  outcomes  (regional  security  or  successful 
peacekeeping  deployments).  Understanding  these  linkages  should  help  the  program 
manager  direct  future  investment  as  well  as  explaining  and  defending  the  value  of  the 
GPOI  to  resource  sponsors. 

Achieving  this  broader  understanding,  to  include  linkages  between  elements,  can 
be  better  appreciated  by  viewing  the  GPOI  system  as  a  process;  by  thinking  of  the  system 
as  a  process,  defined  measurement  points  are  offered.  As  shown  in  Figure  2,  this  research 
suggests  binning  the  GPOI  process  into  five  categories:  plans  &  policy,  inputs,  processes, 
outputs,  and  outcomes.  What  is  required  is  a  broader  assessments  framework  to  manage 
the  spectrum  of  data,  as  well  as  to  manage  additional  assessment  mechanisms. 


Figure  2.  Viewing  the  GPOI  System  as  a  Continuous  Process  (after  Santos  2011) 


This  is  principally  a  data  aggregation  issue.  Different  parts  of  the  GPOI  program 
are  likely  making  useful  system  measurements,  but  the  information  is  not  necessarily 
being  captured  by  the  assessments  framework.  For  example,  the  State  Department 
comptroller  is  no  doubt  capturing  the  financial  expenditures;  however,  that  data  needs  to 
be  sent  to  a  central  node  that  is  managing  the  assessment  framework. 

Hence  the  second  piece  of  holistic  thinking,  which  is  to  view  the  solution  space 
both  in  terms  of  the  technical  and  non-technical.  The  technical  element  of  the  solution 
refers  to  developing  the  right  assessment  models  (what  this  thesis  refers  to  as  the  data- 
based  architecture).  The  non-technical  element  refers  largely  to  significant  inter/intra 
agency  coordination  (what  this  thesis  refers  to  as  the  organizational-based  architecture) 
that  is  necessary  to  actualize  any  assessment  framework.  The  organizational  element 
became  prominent  in  the  solution  design  when  it  was  apparent  that  additional  funds  for 
assessment  were  likely  minimal;  hence,  this  thesis  advocates  building  a  cooperative 
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assessments  framework  to  the  greatest  degree  versus  building  a  “better  stove  pipe”  that 
only  one  node  in  the  system  can  execute. 

This  leads  into  the  third  main  recommendation,  which  is  to  develop  a  coherent 
assessment  framework  from  a  systems  architecting  approach.  The  general  approach,  as 
illustrated  in  Figure  3,  is  adapted  from  a  business  architecture  developed  by  Totem  Ltd,  a 
New  Zealand-based  IT  firm  (Totem  Ltd.  2011).  The  adapted  framework  can  be 
understood  by  breaking  it  into  three  levels:  structured  solution  space,  structured  problem 
space,  and  actualizing  element  space. 

The  heart  of  the  structured  solution  space  is  a  capabilities-based  architecture  that 
states  the  problem  (as  articulated  using  Systems  Engineering  methods)  in  terms  of 
required  capabilities.  The  capabilities-based  architecture  is  then  achieved  by  a  functional 
architecture.  Since  the  solution  is  naturally  information  intensive  as  well  as  complicated 
in  its  implementation,  the  functional  architecture  is  then  bifurcated  into  a  supporting  data- 
based  architecture  as  well  as  into  an  organizational  architecture.  From  the  data  and 
organizational  architectures,  the  actualizing  elements  (e.g.,  assessment  models)  can  then 
be  derived  in  a  traceable  manner.  Additionally,  by  developing  a  coherent  framework, 
other  elements,  such  as  organizational  components  can  be  worked  into  the  solution  space. 
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Figure  3.  Proposed  Assessments  Framework  (after  Totem  Ltd.  2011) 


Follow  on  work  should  focus  on  two  main  lines  of  effort.  The  first  would  be  to 
refine  and  adopt  a  coherent  assessment  framework.  The  second  would  be  to  make  the 
selection  of  specific  metrics  based  on  the  adopted  framework.  The  difficulty  is  not  in 
generating  metrics,  but  in  picking  the  right  ones.  It  is  achievable  using  a  capabilities 
management  approach  as  suggested  in  this  research. 
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I.  INTRODUCTION 


A.  BACKGROUND 

The  United  States  Southern  Command  (USSOUTHCOM)  requested  that  the 
Systems  Engineering  Department  at  the  Naval  Postgraduate  School  conduct  research  on 
their  assessment  methodology  for  the  Global  Peace  Operations  Initiative  (GPOI).  The 
GPOI  is  a  U.S.  government  initiative,  implemented  jointly  by  the  State  Department  and 
Department  of  Defense  that  seeks  to  increase  the  international  peacekeeping  capability  of 
partner  nations  through  training  events,  infrastructure  development,  equipage,  and 
financial  support  (Bureau  of  Political-Military  Affairs  and  Office  of  Plans,  Policy  and 
Analysis  2013).  Accurately  and  consistently  measuring  the  impact  of  the  various  GPOI 
activities  has  been  the  principal  challenge  in  the  assessment  process.  The  GPOI  program 
managers  require  a  feedback  mechanism  in  order  to  evaluate  past  commitments  and  to 
help  guide  future  investments.  From  the  standpoint  of  military  operations,  the  lack  of 
successful  assessment  makes  it  difficult  for  the  combatant  commands  to  understand  how 
GPOI  efforts  impact  their  particular  lines  of  effort  as  part  of  their  broader  theater 
campaign  plan. 

The  principal  research  effort  as  proposed  is  to  develop  a  GPOI  assessment 
framework  using  systems  engineering  theory  with  the  intent  of  building  an  architecture 
that  outputs  not  just  correct  information,  but  the  right  information.  With  sound 
assessment  architecture  in  place,  a  series  of  models  are  then  developed  in  support  of  the 
overarching  framework.  The  effort  is  first  grounded  in  a  stakeholder  analysis  and  a 
mission  definition  phase  to  help  focus  the  problem  space.  A  feedback  loop  with  the 
stakeholder  is  used  to  ensure  that  fundamental  assumptions  and  conclusions  as  to  the 
desired  output  are  sound.  The  assessment  framework  and  the  associated  supporting 
models  are  constructed  by  use  of  systems  engineering  concepts  such  as  decomposability, 
traceability,  systems  architecting  principles,  and  systems  modeling  and  analysis. 

After  a  first  order  analysis,  it  is  suggested  that  the  current  GPOI  assessment  could 
potentially  be  improved  by  the  application  of  systems  thinking.  In  systems  engineering 


I 


terms  the  current  framework  is  a  verification  tool,  in  that  it  seeks  to  answer  the  question: 
“Was  the  money  spent  as  directed?”  While  verification  is  an  important  and  necessary 
component,  it  cannot  by  itself  provide  the  level  of  insight  that  is  actually  required.  An 
assessment  framework  that  is  both  verification  and  validation  is  needed.  Validation 
answers  the  questions:  “Given  that  a  solution  was  reached  (or  a  course  of  action  taken), 
was  it  the  right  one?”  For  example,  if  the  GPOI  makes  a  financial  investment  in  a  partner 
nation  to  build  a  schoolhouse  to  train  its  peace  keeping  units,  a  sound  assessment 
framework  would  address  the  verification  (arc  of  money  from  GPOI  to  a  finished  school 
house)  as  well  as  the  more  ultimate  validation  component  (did  the  school  house  as  built 
improve  unit  readiness  or  the  nation’s  ability  to  increase  capacity  in  a  mission  area?). 

B.  RESEARCH  QUESTIONS 

1.  Primary  Research  Question 

The  primary  research  question  of  this  thesis  is:  can  the  complexity  of  assessing 
the  Global  Peace  Operations  Initiative  be  understood  and  managed?  Another  way  of 
stating  this  question  is:  can  structure  be  lent  to  an  otherwise  ill-defined  problem  space? 

2.  Secondary  Research  Question 

The  secondary  research  question  of  this  thesis  is:  Given  a  structured  problem 
space,  how  should  the  supporting  assessment  models  be  built?  The  crux  of  this  question 
is  what  specifically  should  be  measured  from  the  system  and  why.  There  are  in  existence 
both  explicit  and  implicit  measurement  models  that  pertain  to  GPOI,  but  which  of  them 
are  valid? 

C.  OBJECTIVES 

In  support  of  the  research  questions  a  series  of  objectives  are  stated.  These 
objectives  represent  how  the  potential  answers  to  the  research  questions  would  likely  be 
actualized. 
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1.  Expressing  the  GPOI  as  a  System 

The  first  step  is  to  clearly  articulate  the  problem  and  lay  out  the  requirements  for 
an  assessment  system  that  will  meet  the  stakeholder  needs.  The  emphasis  here  is  on  the 
problem  space,  the  focus  should  be  on  determining  what  the  system  needs  to  achieve  and 
not  on  how  the  system  is  to  achieve  it.  Classical  SE  (Systems  Engineering)  methods  such 
as  stakeholder  and  functional  analysis  will  be  employed.  This  objective  is  addressed  in 
Chapter  III. 

2.  Developing  a  GPOI  Assessments  Framework 

In  this  effort,  the  principles  of  systems  architecting  are  employed  with  the 
intention  of  helping  the  sponsor  manage  the  complexity  represented  in  GPOI  assessments 
by  offering  an  overarching  framework.  Eor  the  purposes  of  this  research,  the  framework 
refers  to  the  overall  assessment  architecture  that  connects  all  of  the  various  functions  and 
operational  activities  necessary  to  pull  the  required  information  from  the  system.  The 
assessment  framework  is  a  much  broader  view  that  looks  beyond  just  describing  what 
should  be  measured  (e.g.,  a  specific  assessment  model)  as  it  takes  into  account  broader 
concerns  such  as  who  is  measuring,  were  the  measurements  are  taken,  and  how  the 
measurements  are  extracted.  By  starting  with  an  architecting  approach,  it  is  hoped  that 
the  subsequent  assessment  models  that  are  offered  will  be  able  to  operate  properly  within 
the  broader  organizational  construct  from  which  they  will  be  employed.  This  objective  is 
addressed  in  Chapter  IV. 

3.  Developing  Supporting  Assessment  Models 

In  order  to  fill  out  the  aforementioned  assessment  framework  a  series  of 
supporting  assessment  models  are  required.  A  series  of  measurement  models  can  be 
derived  using  systems  engineering  theory.  In  the  final  analysis  it  is  possible  that  a  mix  of 
existing,  modified  existing,  and  new  models  will  be  offered  as  the  suggested  composition 
of  an  assessments  framework.  This  objective  is  addressed  in  Chapter  V. 
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4.  Recommendations  for  the  GPOI  Assessments  Program 

Based  on  the  findings  of  the  research  a  series  of  specific  recommendations  are 
made  to  improve  the  GPOI  assessment  methodology.  The  final  check  on  whether  a  given 
recommendation  potentially  offers  value  is  an  evaluation  of  whether  it  is  likely  to  aide 
improved  decision  making.  After  all,  the  end  state  is  not  information  for  information 
sake,  but  it  is  the  right  information  for  the  right  question  to  aid  analysis  and  to  ultimately 
assist  in  the  making  of  sound  decisions.  This  objective  is  addressed  in  Chapter  VI. 

D.  METHODOLOGY 

There  are  two  mutually  supportive  processes  that  have  contributed  to  this 
research:  the  individual  thesis  methodology  and  the  overarching  project  methodology. 
The  methodology  of  this  thesis  refers  to  the  set  of  processes  that  were  employed  towards 
achieving  the  objectives  as  previously  enumerated.  The  second  methodology  is  the 
overall  research  plan  that  encompasses  multiple  researchers  working  on  the  same  general 
problem,  GPOI  assessments,  but  from  different  angles  and  with  different  focuses. 

Other  mutually  supportive  work  is  either  being  conducted  concurrently  or  is 
notionally  planned  for  future  assignment.  Understanding  this  thesis’s  role  within  the 
larger  context,  offers  two  important  advantages.  The  first  benefit  is  that  it  is  hoped  that 
the  intended  work  of  others,  either  that  which  is  conducted  in  parallel  or  follow  on 
efforts,  would  aid  in  the  proper  scoping  and  focus  of  this  thesis’s  research  to  the 
maximum  benefit  of  the  research  sponsor.  Secondly,  the  parallel  efforts  may  offer 
insights  along  the  way,  which  might  further  improve  the  research.  For  further  discussion 
of  the  overarching  GPOI  research  project  see  Appendix  A. 

The  methodology  of  this  thesis  is  explained  by  articulating  the  architecting 
approach  and  the  product  development.  For  the  purposes  of  this  thesis,  the  architecting 
approach  refers  to  how  the  problem  is  framed  while  the  systems  engineering  practices 
refer  to  the  methods  that  are  employed  to  process  information  and  actualize  results  that 
are  discussed  in  greater  detail  in  Chapter  II.  Since  this  effort  is  fundamentally  time 
constrained,  it  must  be  executed  on  a  schedule  with  a  focus  on  product  development, 
hence  the  necessary  programmatic  element. 
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1.  Systems  Architecting 

The  end  state  deliverables  of  both  this  thesis  and  the  larger  research  effort  are 
generally  characterized  as  both  systems  engineering  or  modeling  and  simulation 
products.  However,  it  is  believed  that  the  before  focused  SE  methods  are  applied,  a 
degree  of  systems  architecting  must  be  undergone  first. 

By  making  systems  architecting  an  explicit  element  in  the  design  process,  is  to 
admit  that  there  is  a  gulf  between  the  user,  who  is  or  whose  problems  are  generally  non¬ 
technical,  and  the  engineer  whose  is  and  whose  products  are  generally  technical.  Maier 
and  Rechtin  (2009,  xvii)  in  The  Art  of  Systems  Architecting  succinctly  state: 

Architecting  embraces  the  world  of  the  user/sponsor/client,  with  all  the 
ambiguity  and  imprecision  that  may  entail.  Architecting  seeks  to 
communicate  across  the  gap  from  the  user/sponsor/client  to  the 
engineer/developer,  and  architecting  is  complete  (at  least  in  its  initial 
phase)  when  a  system  is  well-enough  defined  to  engage  developers. 

It  is  believed  that  before  specific  measurements  models  are  crafted  that  a  broader 
vision  of  the  GPOI  assessment  system  must  be  formulated.  The  architecture  in  this  case 
would  go  beyond  physical  abstractions  and  would  include  elements  such  as 
organizational  behavior  and  the  stakeholders’  priorities. 

2.  Product  Development 

The  principal  lines  of  effort  of  this  thesis  are  decidedly  characterized  as  applied 
research  and  not  as  basic  research.  It  is  of  course  hoped  that  the  unique  application  of 
systems  engineering  theory  to  the  problem  of  operational  assessments  will  contribute  to  a 
larger  body  of  knowledge  that  is  generalizable  beyond  the  test  case  of  GPOI.  Thus  the 
end  state  is  primarily  product  development  and  is  not  theory  expansion. 

The  products  contained  within  are  proffered  to  be  of  two  general  families: 
prescriptive  products  and  descriptive  products.  Descriptive  products  describe  the  system 
and  benefit  the  user  by  giving  him  or  her  greater  understanding  to  include  increased 
appreciation  of  assumptions,  context,  universe  of  the  possible,  limiting  reagents  in  a 
process,  and  so  on.  Prescriptive  products  are  those  that  offer  substantive  solutions  or 

suggested  improvements.  While  stakeholders,  agnostic  of  the  problem,  will  generally  be 
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seeking  prescriptive  products,  the  descriptive  might  be  as  important,  if  not  more 
important,  as  they  enable  the  refinement  of  requirements.  It  is  axiomatic  to  state,  but  of 
course  starting  with  the  right  problem  in  order  to  find  the  right  solution,  is  perhaps  the 
most  fundamental  and  practical  application  of  systems  engineering  theory.  Figure  1 
represents  at  a  high  level  the  product  families  developed  by  this  thesis 


GPOl  ASSESSMENT  THESIS  PRODUCT  FAMILY 


Figure  1.  GPOI  Assessment  Product  Family 


To  achieve  the  state  ends  of  this  thesis  three  lines  of  effort  are  employed 
throughout:  problem  understanding,  systems  architecting,  and  model  development  (see 
Figure  2).  As  previously  mentioned,  while  problem  understanding  is  the  foundational 
effort,  it  does  not  stop  at  the  systems  architecting  phase.  The  intent  is  for  the  discovery 
process  to  continually  add  understanding  to  the  original  problem.  Likewise,  while  the 
systems  architecting  is  foundational  to  the  model  development,  the  architecting  continues 
throughout  as  the  development  of  the  models  will  in  turn  influence  and  better  define  the 
original  architecture. 
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Figure  2.  Thesis  Product  Development 


E.  INITIAL  PROBLEM  DEFINITION 

As  stated  earlier,  the  current  GPOI  assessment  program  is  verification  and  not  a 
validation  mechanism.  While  this  is  still  held  to  be  the  central  deficiency  of  the  current 
assessment  model  as  it  is  understood,  it  is  a  symptom  versus  the  root  cause  (Driscoll 
2011,  27-32).  However,  it  is  proffered  that  the  principal  trouble  lies  in  the  inherent 
difficulty  in  thinking  about  the  problem  in  a  holistic  manner.  That  is,  the  GPOI  program 
is  such  a  large  and  complicated  system,  that  in  order  to  properly  measure  it,  the  assessors 
must  be  able  to  widen  their  collective  aperture.  Of  course,  when  the  breadth  and  depth  of 
analysis  is  expanded  the  additional  problem  of  complexity  is  introduced.  Herein  lies  what 
is  believed  to  be  the  root  problems  facing  GPOI  assessments:  the  sheer  scale  and 
intricacy  of  the  Global  Peace  Operations  Initiative  exceeds  the  ability  of  the  current 
assessment  process  to  manage  complexity. 

There  are  perhaps  other  indicators  that  assessments  program  requires 
improvement  such  as  a  lack  of  a  codified  assessment  process  and  unclearly  defined 
organizational  assessment  roles.  However,  these  are  all  symptoms  of  the  lack  of  system 
level  thinking  being  directed  towards  the  problem.  Thus,  if  the  problem  is  not  approached 
properly,  then  the  various  elements  of  the  solution  space  (such  as  the  aforementioned 
process  and  appropriate  organizational  dynamics)  will  not  develop  satisfactorily. 
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F.  PROPOSED  SOLUTION 

If  the  fundamental  problem  is  the  mode  of  thinking,  then  the  solution  design  starts 
there.  As  Patrick  Driscoll  (2011)  stated  neatly,  “The  most  powerful  way  to  improve  the 
quality  of  your  results  is  to  improve  the  way  you  think”  (2011,  28).  Thus  the  first  step  is 
to  recognize  the  GPOI  for  what  it  is,  a  highly  intricate  and  ever  evolving  system  and  then 
to  embrace  the  imperative  for  understanding  and  managing  that  complexity  if  a  robust 
assessment  system  is  ever  to  be  developed  and  employed  successfully. 

While  the  end  users  likely,  and  rightly,  desire  sound  assessment  models  so  that 
they  would  know  what  to  measure  and  what  the  measurements  mean,  that  is  the 
fundamentally  wrong  level  of  abstraction  to  start  with.  The  first  step  is  to  architect  a 
framework  that  accounts  for  the  entirety  of  the  “assessment  system.”  While  the 
framework  includes  the  assessment  models  themselves,  it  must  also  incorporate  other 
elements  such  as  organizations/agencies,  supporting  operational  activities,  and  external 
but  influential  actors,  to  name  a  few.  The  benefit  of  the  architecting  approach  is  that  it 
will  hopefully  help  determine  the  pertinent  requirements  beyond  what  is  required  in  the 
assessment  models  themselves.  These  other  requirements  will  likely  include,  but  not  be 
limited  to,  inter/intra-organizational  coordination  necessities,  or  codified  processes.  As 
these  non-physical  elements  of  the  architecture  will  likely  drive  the  success  or  failure  of 
the  system,  they  are  probably  as,  if  not  more,  important  than  the  assessment  models 
themselves. 

While  the  solution  space  is  broad,  any  effective  resolution  must  offer  both 
physical  design  (models)  and  organizational/behavioral  design.  The  degree  of 
overarching  framework  that  is  taken  on  will  influence  the  models.  That  is,  if  the  end  users 
support,  say  a  collaborative  data  collection  framework,  then  each  node  in  the  system 
(such  as  a  nation,  agency,  or  an  office)  will  have  fundamentally  different  models  than  if 
actors  within  the  system  are  acting  in  a  more  independent  nature. 
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II.  THEORETICAL  ERAMEWORK  AND  LITERATURE 

REVIEW 


The  intent  of  this  chapter  is  twofold.  First  it  is  to  articulate  the  elements  of 
systems  engineering  theory  that  were  applied  in  the  course  of  the  research  to  arrive  at  the 
proposed  solutions  of  this  thesis.  Secondly,  and  perhaps  more  importantly,  it  is  to  show 
that  there  is  value  in  viewing  the  GPOI  as  a  system  and  that  systems  thinking  can 
effectively  be  applied  to  follow  on  efforts. 

A.  SYSTEMS  ENGINEERING 

What  follows  is  a  discussion  on  fundamental  systems  engineering  theory  and  its 
applicability  in  viewing,  thinking  about,  and  designing  solutions  for  the  GPOI 
Assessments  problem.  The  intent  is  not  to  write  an  authoritative  account  on  the  universe 
of  systems  engineering,  rather  the  aim  is  to  highlight  the  portions  that  are  believed  to  be 
directly  applicable  towards  understanding  the  relevant  problem  space  of  the  GPOI  and 
developing  sound  solutions  in  the  area  of  assessments. 

I.  Systems  Thinking 

Making  the  leap  that  a  given  problem  set  would  benefit  from  systems  thinking 
requires  that  one  assumption  be  made,  one  reality  be  understood,  and  one  imperative  be 
embraced.  The  assumption  is  that  GPOI  can  be  viewed  as  a  system.  An  inherent  attribute 
of  many  systems,  to  most  assuredly  include  the  GPOI,  is  one  of  extreme  complexity.  And 
finally,  the  imperative  to  be  embraced,  given  that  the  problem  is  described  as  a  system 
and  very  intricate  one  at  that,  is  that  in  order  to  engineer  and  realize  a  solution  set,  the 
designer  must  have  some  capacity  to  think  holistically  about  the  problem  space  and  then 
an  ability  to  manage  said  complexity  (Driscoll  2011). 

a.  What  Is  a  System,  and  Is  GPOI  a  System  ? 

Understanding  what  a  system  is  and  whether  or  not  the  Global  Peace  Operations 
Initiative  can  be  viewed  as  one  is  a  necessary,  if  albeit  an  unexciting,  prerequisite  to 
applying  systems  engineering  theory  to  the  problem  space. 
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In  Decision  Making  in  Systems  Engineering  and  Management  Patrick  Driscoll 
(2011)  provides  a  clear  definition  of  the  concept  of  the  system.  He  describes  a  system  as 
“an  integrated  set  of  elements  that  accomplish  a  defined  objective”  (2011,  33).  GPOI  fits 
this  definition  as  it  does  in  fact  readily  appear  to  be  an  integrated  set  of  elements  (i.e., 
partner  nations,  funding  streams,  and  training  activities)  that  act  in  concert  to  accomplish 
a  defined  set  of  objectives  (i.e.,  peace  keeping  capacity,  goodwill,  and  regional  security). 

Expounding  on  the  fundamental  definition  Driscoll  (2011)  maintains  that  the 
“operation  of  the  system  lies  at  the  heart  of  the  decision  to  be  made  and  not  the  system 
itself’  (2011,  28).  This  emphasis  is  most  useful  for  the  GPOI  problem  as  it  focuses  the 
analysis  on  what  the  system  does,  or  should  do,  and  not  on  the  makeup  of  its  constituent 
elements.  This  orientation  towards  purpose  and  not  structure  is  useful  in  system 
architecting,  which  will  be  a  foundational  effort. 

b.  Systems  Taxonomy  and  the  GPOI 

Patrick  Driscoll’s  (2011)  systems  taxonomy  of  physical,  abstract,  or 
unperceivable,  was  used  for  its  ability  to  articulate  the  defining  attributes  of  the  GPOI 
system.  A  physical  system  is  any  system  that  has  observable  physical  interactions  in  time 
and  space  (such  as  aircraft  or  factory  machinery).  An  abstract  system  is  the  connection  of 
concepts  that  may  not  be  observable  and  but  can  usually  be  readily  substantiated 
(management  plans  or  public  policy).  Finally,  there  are  unperceivable  systems  whose 
principal  characteristic  is  their  inability  to  have  their  constituent  elements  and  associated 
interactions  fully  observed  and  understood.  Using  the  U.S.  economy  as  an  example, 
Driscoll  (2011)  illustrates  the  common  challenge  in  dealing  with  the  inherent  complexity 
of  an  unperceivable  system  by  observing  that  “despite  technology  advances  and  Nobel 
laureate  awardees  in  economics,  error- free  future  state  forecasts  of  this  system  remain 
impossible  to  attain”  (2011,  35). 

The  GPOI  certainly  contains  subsystems  that  are  physical  (training  facilities  and 
IT  networks),  and  it  certainly  has  many  characteristics  of  an  abstract  system  (funding 
approval  plans  and  memorandums  of  understanding).  However,  it  is  best  described  as  an 
unperceivable  system. 
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c. 


Implications  of  Dealing  with  an  Unperceivable  System 


The  importance  of  accepting  the  GPOI  as  an  unperceivable  system  is  that  it  sets 
realistic  expectations.  The  sheer  scale,  complexity,  and  sometimes  hidden  nature  make  it 
impossible  to  capture  all  of  the  constituent  elements  and  their  relationships.  Thus,  any 
model  of  the  GPOI  system  will  always  be  an  imperfect  approximation.  An  imperfect 
model  will  not  generate  perfect  conclusions  every  time. 

This  is  not  to  say  that  the  enormity  of  the  challenge  should  necessitate  abandoning 
the  project.  While  full  and  complete  understanding  is  unattainable,  very  useful  partial 
understanding  is  attainable.  Going  back  to  the  example  of  the  U.S.  economy,  while 
officials  working  at  the  Treasury  and  the  Federal  Reserve  cannot  know  with  one  hundred 
percent  certainty  the  impact  of  their  actions  (because  they  are  dealing  with  an 
unperceivable  system),  their  understanding  of  the  model,  albeit  limited,  gives  them  a 
basis  for  making  rational  decisions  aimed  at  specified  end  states.  Thus  the  mandate  for 
the  designer  of  the  GPOI  assessments  is  to  build  the  highest  fidelity  framework  possible 
that  can  provide  useful  insights  to  decision-makers  and  then  properly  caveat  its  inherent 
limitations. 

In  addition  to  setting  expectations  properly,  recognizing  the  GPOI  as  an 
unperceivable  system  naturally  directs  one  towards  certain  approaches.  If  part  of  the 
fundamental  problem  is  complexity  then  part  of  the  solution  (including  the  solution 
process)  should  be  simplification.  As  addressed  later,  one  of  the  guiding  architecting  and 
modeling  principles  is  that  of  simplification  wherever  possible. 

d.  System-Level  Solutions  vice  Symptom-Level  Solutions 

Systems  thinking,  as  it  pertains  to  engineering  design,  is  first  and  foremost  an 
orientation  towards  the  system’s  required  objective  set  (Driscoll  2011).  Objectives,  in  this 
case,  would  be  defined  as  what  the  stakeholder  set  has  determined  as  what  the  system 
needs  to  achieve.  Systems  thinking  proposes  beginning  with  the  end  state  of  the  system 
and  then  building  backwards  to  include  the  underlying  elements  (functions,  operational 
activities,  structure,  and  components)  that  are  needed  to  achieve  the  stated  objectives. 
The  procedural  benefit  of  focusing  on  the  objectives  is  that  both  the  requirement 
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development  and  objective  articulation  are  given  the  proper  priority,  thus  a  measureable 
arc  from  fundamental  problem  to  fundamental  end  state  can  be  drawn. 

A  temptation  in  system  design  or  system  analysis  may  be  to  focus  on  an  element 
of  the  problem  set  that  is  the  most  visible;  however  the  risk  is  that  the  engineer  may  have 
emphasized  what  is  symptomatic  and  not  systemic.  While  there  may  be  good  reasons  to 
address  symptoms,  not  the  least  would  be  the  potential  for  “low  hanging  fruit”  that 
promises  immediate  value  added  for  a  reasonable  investment.  However,  this  “problem 
chasing”  paradigm  should  not  be  the  fundamental  methodology  applied  in  system 
analysis  and  design.  Hence  when  conducting  system  analysis  the  root  causes  should  be 
sought  and  when  conducting  system  design  the  end  state  should  always  remain  at  the 
fore.  The  benefit  of  this  line  of  thought,  as  summarized  by  Patrick  Driscoll  (2011)  is  that, 
“This  natural  focus  on  output  (i.e.,  results,  effects)  provided  by  systems  thinking  creates  a 
goal-oriented  frame  of  reference  that  produces  long-term,  effective  system-level  solutions 
rather  than  short-term,  symptom-level  solutions  [emphasis  added]”  (2011,  28). 

Systems  level  thinking  can  be  applied  to  the  problem  of  developing  a  sound  GPOI 
assessments  architecture.  It  demands  first  that  the  objectives  of  the  GPOI  assessments  be 
clearly  articulated.  Later  chapters  establish  that  these  objectives  are  in  fact  well  defined 
but  they  have  not  been  made  the  foundation  of  the  current  assessment  design.  Using  the 
GPOI  objectives  as  stated,  the  supporting  functions  and  processes  can  be  enumerated, 
thus  creating  the  supporting  architectures  from  which  it  is  proposed  the  assessment 
framework  be  built  (see  Chapter  IV). 

2.  Problem  Definition 

The  beginning  is  the  most  important  part  of  the  work. 

Plato,  4*  century  B.C. 

As  previously  mentioned,  defining  the  problem  properly  is  the  foundation  of  the 
entire  systems  engineering  process.  The  end  state  is  to  have  a  solution  that  addresses  the 
stated  problem.  This  sounds  so  axiomatic  so  as  to  not  warrant  mentioning,  however  it 
quite  possible  to  engineer  a  “great”  solution  only  to  find  it  does  not  address  the  real 
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problem.  Hence,  proper  emphasis  is  placed  on  exploring  the  problem  space.  As  Albert 
Einstein  famously  quipped,  “If  I  were  given  one  hour  to  save  the  planet,  I  would  spend  59 
minutes  defining  the  problem  and  one  minute  resolving  it”  (Spradlin  2012,  84). 

a.  Problem  Definition  Processes 

This  research  developed  the  problem  statement  using  two  main  families  of  effort, 
stakeholder  analysis  and  functional  analysis.  The  end  state  of  stakeholder  analysis  is  to 
gain  an  appreciation  for  what  the  customer  or  end-users  ultimately  require  (Buede  2000). 
Stakeholders  are  defined  here  in  this  thesis  as  principally  those  entities  that  are  likely  to 
be  directly  involved  in  either  the  production  of  GPOI  assessments  and/or  the 
consumption  of  GPOI  assessments. 

The  end  state  of  functional  analysis  is  to  develop  the  capabilities  that  will 
ultimately  need  to  address  the  problem  (Trainor  and  Parnell  2011).  This  thesis  used  a 
functional  hierarchy  to  enumerate  and  organize  the  required  functions  of  any  proposed 
solution.  The  key  attribute  of  a  functional  analysis  is  that  it  is  fundamentally  solution 
agnostic;  that  is,  it  addresses  what  needs  to  be  done,  not  how  it  needs  to  be  done. 

b.  Problem  Articulation 

After  a  foundational  research,  stakeholder  analysis,  and  functional  analysis  have 
been  conducted  the  final  step  is  to  articulate  the  problem.  An  effective  problem 
articulation  includes  properly  scoping  and  bounding  the  problem  space.  Additionally,  any 
limitations  or  assumptions  should  be  clearly  stated.  As  previously  mentioned,  based  on 
the  system  complexity  and  lack  of  observable  data  in  some  cases,  making  well-reasoned, 
and  well-articulated,  assumptions  are  particularly  important. 

As  the  research  progresses  and  new  understanding  is  gained,  either  as  a  result  of 
the  discovery  process  or  interaction  with  the  stakeholders,  the  problem  statement  will 
likely  require  updating.  Trainor  and  Parnell  (2011),  in  Decision  Making  in  Systems 
Engineering  and  Management  noted,  “The  initial  problem  is  never  the  real  problem” 
(2011,300). 


13 


Developing  and  expressing  a  problem  statement,  is  considered  both  a  means  and 
an  end.  It  is  a  means  in  that  it  is  a  necessary  prerequisite  to  develop  a  rational  solution, 
but  it  is  also  an  end  as  descriptive  understanding  can  itself  be  useful.  The  development  of 
the  problem  statement  for  the  assessment  of  the  Global  Peace  Operations  Initiative  is 
detailed  in  Chapter  III. 

3.  Systems  Architecting 

A  systems  architecting  approach  has  been  employed  to  offer  a  proposed  GPOI 
assessments  framework.  As  already  mentioned,  parsing  the  GPOI  is  a  difficult  challenge 
based  on  the  ill-defined  nature  of  the  problem  as  well  as  the  high  level  of  complexity 
inherent  in  the  system.  Architecting,  for  the  purposes  of  this  thesis,  is  understood  as  the 
approach  used  to  conceptualize  the  structure  and  behavior  of  the  system.  What  follows  is 
brief  theoretical  overview  of  the  systems  architecting  approached  taken  in  Chapter  IV  to 
develop  a  proposed  assessments  framework  for  the  Global  Peace  Operations  Initiative. 

a.  Why  the  GPOI  Assessments  Problem  Benefits  from  Systems 
Architecting 

Traditional  engineering  methods  cannot  be  effectively  applied  if  the  problem  is 
not  well  enough  defined.  Addressing  the  ambiguity  of  the  GPOI  system  and  attempt  to 
give  it  a  working  structure  is  essential.  Assessment  models  can  be  engineered,  but  not 
until  the  requirement  space  is  sufficiently  understood.  Thus,  systems  architecting  is 
fundamentally  an  effort  of  translation  from  the  context  of  the  user  to  the  more  ordered 
world  of  the  engineer.  The  sheer  size  and  convolution  of  the  system  may  indicate,  as 
Maier  and  Rechtin  (2009)  point  out  that  “different  problem-solving  techniques  are 
required  at  high  levels  of  complexity  than  at  low  ones.  Purely  analytical  techniques, 
powerful  for  the  lower  levels,  can  be  overwhelmed  at  the  higher  ones”  (2009,  6). 

b.  Architecting  Approach 

The  general  approach  to  handling  an  intensely  complicated  problem  is  to  try  to 
simplify  it  by  focusing  on  the  most  important  elements.  Of  the  several  approaches  to 
architecting  that  can  be  taken  in  order  to  begin  conceptualizing  the  system,  Dennis 
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Buede’s  (2000)  approach  has  been  adopted.  According  to  Buede  (2000),  there  are  three 
fundamental  phases  that  are  helpful  in  fleshing  out  a  working  architecture:  functional 
architectural  development,  physical  architectural  development,  and  operational 
architectural  development  (Buede  2000). 

The  functional  architecture  intent  defines  what  a  system  must  do,  usually  from  a 
top  down  orientation.  The  physical  architecture  consists  of  the  “resources”  or  components 
that  are  capable  of  implementing  or  actualize  functions  (Buede  2000).  By  reconciling  the 
physical  and  functional  architectures,  an  operational  architecture  is  created  that  acts  as 
the  beginning  of  a  system  concept. 

4.  Systems  Modeling  and  Analysis 

Throughout  this  thesis  a  series  of  models  are  used  to  express  the  GPOI  as  a 
system.  The  modeling  effort  is  a  bit  of  a  translation  challenge,  as  the  GPOI,  which  is  not 
a  classical  engineering-physics  based  system,  must  be  represented  by  systems  models  that 
are  inherently  mathematical  abstractions.  The  importance  of  development  mathematical 
models  cannot  be  understated.  As  Blanchard  and  Fabrycky  (2011)  aptly  point  out,  “The 
mathematical  model  readily  indicated  the  type  of  data  that  should  be  collected  to  deal 
with  the  problem  in  a  quantitative  manner,”  which  is  of  course  the  desired  end  state  of 
this  research  project  writ  large  (2011,  1 19). 

The  benefits  are  worth  the  work,  as  sound  models  go  a  long  ways  towards 
achieving  system  understanding  and  lay  the  groundwork  for  future  analytic  efforts  that 
hopefully  can  suggest  substantive  system  improvements.  What  follows  below  is  a  brief 
explanation  of  how  modeling  is  believed  to  apply  to  GPOI  problem  and  some  guiding 
principles  that  were  used  in  development  of  various  models. 

a.  What  Is  a  Model? 

In  the  most  fundamental  terms,  according  to  West,  Kobza,  and  Goerger  (2011)  a 
model  is  “an  abstract  representation  of  a  system”  (2011,  99).  The  more  useful  definition, 
as  it  applies  to  GPOI  assessments,  is  that  a  model  should  be  thought  of  as  a  representation 
of  a  given  system  that  seeks  to  express  the  important  elements,  attributes,  and 


15 


relationships  of  that  same  system.  George  Box  famously  summarized,  “Essentially  all 
models  are  wrong,  but  some  are  useful”  (Champkin  n.d.).  This  is  to  say  that  no  model 
will  mirror  every  element  of  the  system  completely  and  accurately  but  can  nonetheless  be 
a  powerful  tool. 

b.  Why  Models  Are  Needed  to  Assess  the  GPOI 

While  general  insight  of  the  GPOI  is  desired,  what  is  specifically  required  is  an 
understanding  as  to  what  needs  to  be  measured.  Modeling  helps  enumerate  and 
understand  these  key  parameters.  The  fundamental  modeling  process  calls  for  identifying 
the  elements  and  the  processes  in  the  system  and  then  defining  their  relationships.  To 
paraphrase  Gary  Langford  (2013),  all  interactions  between  objects  and  processes  can  be 
thought  of  as  falling  into  one  of  the  four  energy,  matter,  material  wealth,  and  information 
(EMMI)  categories  (Langford  2013).  With  the  key  objects  and  processes  articulated  and 
the  nature  of  their  interactions  defined,  a  mathematically  coherent  abstraction  is 
generated. 

While  the  fundamental  modeling  process  highlights  a  list  of  candidates  to 
measure,  it  is  the  follow  on  analytic  effort  that  helps  in  correlation  and  causation 
determination.  Sensitivity  analysis  can  be  conducted  to  help  determine  the  “predictor 
values,”  with  a  sound  and  validated  set  of  models.  This  represents  the  idealized  end  state 
of  this  effort  (which  would  likely  require  follow  on  work  from  this  thesis).  That  is  to 
understand  the  system  so  clearly  that  only  a  minimum  set  of  parameters  requires 
measurement  so  as  to  gain  whatever  understanding  is  desired. 

c.  Key  Modeling  Principles 

There  is  an  abundance  of  literature  on  the  subject  of  what  makes  a  good  model. 
However,  for  the  purpose  of  developing  models  for  GPOI  assessments  three  main 
attributes  (or  measures  of  quality)  are  emphasized.  These  three  are:  (1)  simplicity,  (2) 
fidelity  and  (3)  balance  (West,  Kobza  and  Goerger  2011). 

(1)  Simplicity.  To  the  greatest  extent  possible,  simplicity  will  be  sought 

building  GPOI  models.  Simplicity  in  models  offers  both  practical  and  theoretical 

advantages.  The  simpler  the  model,  generally  the  easier  it  will  be  for  the  end-user  to 
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understand  and  apply  (West,  Kobza  and  Goerger  2011).  Understanding  is  necessary  for 
the  end  user,  both  to  make  constructive  use  of  the  model  as  well  as  to  offer  improvements 
to  it.  Additionally,  the  simpler  the  model,  generally  the  fewer  and  hopefully  more 
reasonable  assumptions  will  be  required.  A  tradeoff  can  exist  between  complexity  and 
veracity;  that  is,  simplicity  can,  but  not  necessarily,  be  reduced  for  increased  model 
precision  or  accuracy.  However,  according  to  the  principle  of  parsimony  (Occam’s 
razor),  whenever  evaluating  equally  true  explanations,  the  simplest  is  the  best  (Heylighen 
1997).  Hence  the  models  in  this  thesis  were  intentionally  developed  to  be  as  simple  as 
possible.  While  it  is  hoped  that  follow  on  work  will  improve  the  quality  of  the  models,  it 
is  believed  that  any  improvements  should  generally  try  to  maintain,  if  not  reduce, 
complexity,  unless  there  is  a  compelling  case  to  be  made  for  the  added  intricacy. 

(2)  Fidelity.  Fidelity,  as  a  measure  of  quality,  is  defined  here  as  how  well  a 
given  abstraction  represents  the  mirrored  system  (West,  Kobza  and  Goerger  2011).  As 
discussed  earlier  it  is  not  practical,  and  certainly  not  possible,  to  achieve  a  perfect 
representation,  but  to  the  extent  possible  the  modeler  should  strive  for  maximum 
accuracy  of  his  or  her  model. 

(3)  Balance.  Balance,  as  a  measure  of  quality,  is  defined  as  how  well  the 
modeler  has  handled  the  natural  tension  that  exists  between  the  aforementioned  qualities 
of  simplicity  and  fidelity  (West,  Kobza  and  Goerger  2011).  The  key  for  the  modeler  is  to 
be  able  to  identify  and  include  only  the  most  important  parameters  in  his  or  her  model, 
thus  a  good  balance  between  simplicity  and  fidelity  can  be  achieved. 

d.  Modeling  Methodology 

The  general  modeling  processing  employed  is  the  methodology  as  expressed  by 
West,  Kobza,  and  Goerger  (2011).  For  further  discussion  on  modeling  process  according 
to  West  et  ah,  can  be  found  in  Appendix  B. 

e.  A  Note  on  Software  and  Modeling  Languages 

While  the  author  generally  endorses  the  use  of  specialized  modeling  software, 
such  as  Vitech’s  CORE,  a  conscious  decision  has  been  made  to  render  final  models  using 
Microsoft  Office  products  wherever  possible.  This  was  done  for  the  practical  reason  of 
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making  them  more  accessible  for  future  edit.  Specialized  software  requires  that  the  end 
users  undergo  a  learning  period  before  employing  the  software  as  well  as  having  to  clear 
the  not  insignificant  barrier  of  obtaining  licenses  and  permissions  prior  to  use  on 
government  networks. 

By  not  employing  specialized  modeling  software  dealing  with  very  complex  data 
sets  becomes  difficult.  That  is  a  software  package  like  CORE  allows  the  engineer  to 
handle  a  large  number  of  elements  and  their  relationships  in  a  flexible  manner.  However, 
as  the  general  architecting  approach  is  to  simplify  and  focus  on  the  key  elements,  this 
seemed  a  reasonable  trade  for  generating  the  models  in  ubiquitous  software  that  will 
hopefully  make  them  a  little  more  accessible  to  both  the  end  users  as  well  as  any  follow 
on  researchers  who  would  attempt  to  modify  them  and  roll  them  into  simulation 
packages. 

Modeling  languages  were  selected  primarily  for  convenience  of  illustrative 
purposes.  FEED  and  IDEFO  paradigms  were  used  for  their  readily  understandable  nature 
in  hierarchical  views  and  process  explanation,  respectively. 

5.  Developing  Systems  Measures 

Chapters  IV  and  V  explain  that  the  system  measures  that  should  be  used  are  those 
parameters  that  are  desired  end  states,  traced  from  the  highest  level  objectives  down  to 
the  individual  tasks.  These,  by  default  when  developed  correctly  will  in  fact  meet  the 
requirements  for  sound  measures  of  effectiveness  (MOE)  and  measures  of  performance 
(MOP)  as  defined  by  standard  SE  practices.  Of  course,  as  is  argued  later  for  the  purposes 
of  GPOI  assessments  it  is  necessary  to  also  collect  measurements  on  the  inputs  and  the 
processes.  What  follows  is  a  brief  discussion  on  the  characteristics  of  valid  MOEs  and 
MOP. 


a.  Measures  of  Performance 

A  MOP  is  principally  a  question  of  verification.  That  is  the  MOP  should  seek  to 
answer  the  question:  is  the  system  doing  what  it  is  was  designed  for?  In  the  context  of 
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GPOI,  the  question  would  generally  take  the  form  of:  did  the  GPOI  program  execute  as  it 
intended  to?  The  answer  to  this  question  can  be  independent  of  the  answer  to  the 
question,  did  what  we  do  matter? 

b.  Measures  of  Effectiveness 

A  MOE  is  principally  a  question  of  validation.  That  is  the  MOE  should  seek  to 
answer  the  question:  is  the  system  accomplishing  its  stated  end-states?  Which  is  another 
way  of  asking,  has  the  right  solution  been  developed  for  the  right  problem?  A  system  can 
potentially  meet  and  exceed  all  MOPs  but  ultimately  fail  its  MOEs. 

c.  MOEs  and  MOPs  with  Respect  to  Each  Other 

To  paraphrase  West  et  al.  (201 1),  who  offer  a  way  of  distinguishing  the  difference 
of  two  measures,  MOPs  are  fundamentally  internal-system  measures  (i.e.,  did  individual 
components  of  the  system  perform  to  the  design  specifications)  while  MOEs  are 
fundamentally  context  specific  measures  (i.e.,  did  the  system  as  a  whole  perform  in  the 
intended  operational  environment?)  (2011,  96-98). 

Another  useful  way  to  look  at  MOEs  and  MOPs  is  to  view  them  in  the  context  of 
overall  system  design.  As  illustrated  in  the  classic  SE  V-model,  MOEs  are  intentionally 
placed  opposite  from  the  user  requirements  and  the  operational  concept,  whereas  the 
MOPs  are  placed  across  from  system  verification.  In  the  V-model,  as  shown  in  Appendix 
C,  the  elements  on  the  right  hand  side  are  intended  to  be  based  on  the  elements  of  the  left 
hand  side.  This  is  the  fundamental  concept  of  traceability,  or  in  other  words  making  sure 
the  right  parameter  is  being  measured. 

6.  Discovery  Process 

If  the  requirement  set  was  known  at  the  outset,  and  believed  to  be  final,  then  a 
primarily  sequential  method  would  perhaps  be  the  most  appropriate  approach.  However, 
the  discovery  of  the  requirements  is  in  large  part  the  end  state  itself,  which  has  in  part 
necessitated  a  more  iterative  approach  to  the  problem.  As  Maier  and  Rechtin  (2009,  23) 
explain,  “as  the  market,  or  operational  environment,  reveals  new  desires,  those  desires  are 
fed  back  into  the  product.”  Eor  GPOI,  the  market  is  represented  by  the  first  order 
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stakeholders  for  whom  this  research  is  intended  and  the  operational  environment  is  the 
GPOI  program  writ  large,  which  as  a  complex  adaptive  system  itself,  is  continually 
evolving.  Thus,  even  if  the  problem  were  to  be  perfectly  understood,  which  it  is  most 
certainly  is  not,  the  nature  of  the  problem  is  continually  changing  requiring  ever  evolving 
and  new  understanding.  To  paraphrase  Vasant  Honavar,  a  complex  adaptive  system  is 
one  that  is  characterized  by  its  complexity  of  its  behavior  that  is  it  itself  the  product 
of  its  many  interactions  of  constituent  elements  at  different  levels  of  organization 
(Honavar  n.d.). 

The  iterative  process  is  common  in  software  intensive  projects.  While  few  lines  of 
codes  are  expected  to  be  written  in  the  course  of  this  thesis,  the  research  product  line  is 
decidedly  “software”  and  not  “hardware.”  The  products  in  development  are  expected  to 
be  a  framework  and  a  series  of  supporting  models.  These  models,  either  at  the  end  of  this 
research,  or  at  the  start  of  the  follow  on  work  are  expected  to  be  further  refined  and  to  be 
executed  inside  a  simulation  package.  Iterative  approaches,  or  simply  stated  a  series  of 
refinements  based  on  discovery,  is  commonly  understood  by  a  spiral  model  such  as  the 
one  commonly  attributed  to  Barry  Boehm  (1988). 

Figure  3  GPOI  Assessment  Model  Development  Process  represents  the  iterative 
approach  that  was  applied  throughout  this  research  based  on  Boehm’s  classic  spiral 
(Boehm  1988).  The  process  started  with  the  stakeholder  presenting  a  needs  statement, 
which  was  then  interpreted  into  a  series  of  systems  engineering  artifacts  that  could  be 
translated  into  an  assessment  model.  The  iterative  piece  occurs  when  the  designer’s  beta, 
or  initial  design,  is  proposed  to  the  user  who  then  offers  feedback  that  results  in  new 
understanding  by  both  the  stakeholder  and  the  engineer.  This  refinement  process  repeats 
with  (hopefully)  ever  increasing  product  quality  as  project  resources  are  expended. 
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Figure  3.  GPOI  Assessment  Model  Development  Process  (after  Boehm  1988) 


The  spiral  process  as  described  will  likely  be  executed  whether  the  designer  is 
conscious  of  the  engineering  process  or  not.  A  certain  degree  of  trial  and  error  and  back 
and  forth  with  the  customer  will  likely  occur,  whether  or  not  the  designer  is  consciously 
operating  with  the  need  to  think  iteratively.  There  are  two  primary  benefits  to  making  the 
process  explicit.  First,  there  is  an  inherent  value  in  any  construct  that  helps  one  organize 
his  or  her  thinking  when  addressing  a  very  complex  problem.  Secondly,  and  perhaps 
most  importantly,  the  spiral  is  a  frank  admission  that  at  the  outset  the  designer  does  not 
understand  the  requirements  so  he  or  she  must  solicit  them  from  the  customer,  but  the 
customer  does  not  know  them  either.  Thus  the  spiral  model  shows  the  way  out  of  the 
circle  by  starting  with  a  series  of  designs  based  on  initial  understanding  and  working  with 
the  customer  to  develop  an  ever  increasing  understanding  of  the  problem. 
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7.  A  Note  on  Solution  Adoption  and  the  Design  Process 

Richard  Balling  (1999)  speaks  to  the  need  of  engaging  the  customer  in  an 
interactive  design  process  in  his  paper  “Design  by  Shopping:  A  New  Paradigm?”  To 
paraphrase  Balling,  developing  the  best  design  (optimization)  only  really  works  when 
objectives  and  constraints  are  well  known,  which  is  rarely  the  case,  and  certainly  not  the 
case  in  the  GPOI  assessments  problem.  Since  driving  to  a  solitary  optimized  point  in  the 
solution  space  is  unachievable,  it  is  suggested  that  a  series  of  “rich  designs”  be  offered 
from  which  the  customer  can  evaluate,  offer  a  practical  means  of  fleshing  out  the  true, 
prioritized  requirements  (Balling  1999). 

While  this  thesis  is  limited  in  both  its  breadth  and  its  depth,  a  variety  of  different 
views  and  abstractions  of  the  systems  were  developed  in  the  hopes  that  if  not  all,  at  least 
some  would  resonate  with  the  customer  either  for  adoption  or  for  further  study.  A  down 
selection  process,  as  Balling  postulates,  is  likely  to  give  the  designer  and  customer  better 
control  over  the  development  vice  a  binary  response  towards  a  unitary  proposal  (Balling 
1999). 

B.  OPERATIONAL  ASSESSMENT:  DOCTRINE  AND  LESSONS  LEARNED 

The  primary  methodology  of  this  thesis  is  to  decompose  and  then  reconstruct  the 
GPOI  assessments  using  systems  engineering  theory;  however,  the  intent  is  not  to  do  so 
in  a  vacuum.  The  difficulty  in  conducting  relevant  operational  assessments  is  not  a  new 
one,  and  many  people  from  different  arenas  have  grappled  with  the  problem  for  many 
years.  The  intent  of  reviewing  operational  assessment  literature  is  to  a  gain  an 
appreciation  for  how  real-world  practitioners  have  approached  the  problem,  albeit  not 
necessarily  from  a  systems  engineering  framework. 

1.  Previous  Application  of  Systems  Engineering  to  Operational 
Assessments 

At  the  79*  Military  Operational  Research  Society  (MORS)  Symposium 

Christopher  Santos  (2011),  a  Navy  commander  and  then  staff  officer  at  USAFRICOM, 

presented  a  very  cogent  application  of  a  systems  approach  to  assessments  (Santos  2011). 

To  paraphrase  this  author’s  overarching  takeaway  from  the  presentation,  the  system  is  for 
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the  user  and  not  the  other  way  around.  Thus  the  assessment  framework  must  be  designed 
in  a  way  that  works  for  the  user,  with  available  mechanisms  and  resources,  and  it  must  be 
relevant  to  application  environment.  Additionally,  the  presentation  offered  several 
practical  insights  as  to  how  to  marry  real  world  operational  assessments  and  basic 
systems  theory.  What  follow  are  some  of  the  key  concepts  from  Santos’s  presentation 
that  bear  mentioning,  as  understood  and  interpreted  by  the  author  (Santos  201 1). 

a.  The  Importance  of  Plans  in  Assessment 

Santos  (2011)  makes  the  point  that  a  given  operational  or  strategic  plan  is  itself  a 
very  important  part  of  the  assessment  process.  If  the  plan  itself  is  sound  and  built  in 
accordance  with  doctrine,  then  the  necessary  linkages  will  be  clearly  articulated.  In  this 
context,  linkage  refers  to  how  the  plan  describes  the  intended  arc  of  the  policy  to  action 
to  intended  effect  and  finally  to  intended  outcome.  In  SE  terms,  this  is  process 
traceability.  The  implications  of  this  notion  is  that  the  plan  itself  must  be  assessed,  both 
to  mine  the  necessary  measurements  that  should  be  used  as  the  activity  unfolds  (e.g., 
activities,  MOEs,  MOPs)  and  if  need  be  to  provide  feedback  when  a  plan  is  lacking  in  its 
articulation  of  assessments  (Santos  2011). 

b.  Viewing  a  System  as  a  Process  and  Selecting  Measurement  Points 

Santos  (2011)  proposed  modeling  a  plan,  or  series  of  activities,  as  a  process  (see 
Eigure  4).  The  general  idea  is  that  every  activity  will  include  some  input,  a  process,  an 
intended  output  and  an  intended  outcome.  The  benefit  of  this  strategy  is  it  enables  the 
assessor  to  manage  the  complexity  of  analyzing  a  large  activity,  or  series  of  activities, 
that  occur  as  part  of  a  very  complicated  system.  Additionally,  this  process  view 
emphasizes  the  key  linkages  (i.e.,  traceability)  that  are  so  important  to  determine  the  right 
measures  to  analyze  (Santos  2011). 

Eor  the  purposes  of  this  thesis  an  output  is  the  immediate  result  or  objective  of  a 

GPOI  activity.  Examples  of  GPOI  outputs  would  be  number  personnel  trained  in  a  “train 

the  trainer  event”  or  the  amount  and  type  equipage  given  to  deploying  peace  keeping 

units.  Outputs  are  measured  by  MOPs  and  are  generally  quantitative  in  nature.  Outcomes 

are  the  end  states  or  the  “so  what”  of  the  program.  An  example  of  a  GPOI  outcome  would 
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be  enhanced  regional  security.  In  comparison  to  outputs,  outcomes  are  generally  broader 
and  can  take  on  a  more  qualitative  nature.  The  outcomes  in  this  thesis  are  assessed  by 
MOEs.  A  proposed  list  of  outputs  and  outcomes  are  discussed  in  Chapter  V. 


Figure  4.  Model  Used  to  Describe  a  Given  System  to  Be  Modeled 

(from  Santos  2011) 

c.  The  Need  for  Data  Variety 

One  of  the  benefits  of  viewing  an  operational  activity  as  a  process,  as  in  Figure  4, 
is  that  clear  and  defined  measurement  points  are  presented.  If  the  assessment  framework 
takes  measurements  at  each  step  of  the  way,  a  more  complete  picture  can  be  built. 

Santos  (2011)  emphasized  this  to  ensure  the  necessary  data  is  on  hand  to  address 
the  analytic  requirements,  which  is  of  course  the  reason  for  data  collection  in  the  first 
place.  Specifically,  Santos  points  out  that  to  answer  a  given  question  usually  requires 
multiple  data  types.  For  example  to  understand  the  impact  of  a  given  GPOI  investment  on 
an  allied  nation’s  peacekeeping  capability,  it  is  likely  that  at  least  three  classes  of  data  are 
needed:  measure  of  input  (MOI)  (e.g.,  seed  money),  MOP  (e.g.,  training  facility  built), 
and  MOE  (e.g.,  strategic  outcome)  (Santos  2011). 

2.  Lessons  from  Recent  Operational  Assessments 

The  intent  of  this  is  to  build  an  assessments  framework  from  SE  theory,  which  has 
its  intrinsic  value  as  defined  methodology  as  well  as  offering  a  new  approach  to  an  old 
problem.  The  proposed  framework  and  associated  models  should  be  tempered  by  “best 
practices”  from  the  field  of  operational  assessments.  These  lessons  learned  or  best 
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practices  offer  the  designer  a  set  of  heuristics.  The  following  is  a  brief  discussion  on  what 
are  believed  to  be  some  of  the  more  salient  observations  from  recent  struggles  in  the  field 
of  operational  assessments. 

a.  Importance  of  Process  and  Procedure 

In  his  2010  paper  from  the  Naval  War  College,  “Effective  Operational 
Assessment  A  Return  to  the  Basics,”  Robert  Michael  (2010)  lays  out  an  illuminating 
critique  on  the  universe  of  doctrine,  process,  and  procedures  that  apply  to  operational 
assessment.  As  Michael  (2010)  describes,  a  recurring  shortfall  in  codified  assessment 
frameworks  is  a  lack  of  emphasis  on  process  and  procedures.  Michael  (2010)  maintains 
that  most  publications  on  the  subject  focus  on  what  needs  to  be  done,  with  little  mention 
for  who  should  do  it  or  how.  The  point  is  taken,  for  the  GPOI  investigation,  in  that  it  is 
likely  not  sufficient  to  enumerate  the  various  assessment  activities  that  are  necessary  to 
achieve  a  coherent  picture.  Due  regard  must  be  given  to  the  entities  (people  and 
organizations)  that  enact  the  models.  With  the  assessor  in  mind  both  the  assessment 
models  themselves  as  well  as  the  codified  processes  and  procedures  are  necessary. 
(Michael  2010) 

b.  Data  Selectivity 

Michael  (2010)  observes  that  assessors  face  the  “the  tendency  to  measure  what 
can  be  measured  vice  what  should  be  measured”  (2010,  5).  It  appears  that  GPOI  is 
perhaps  not  immune  from  this  tendency  as  there  have  reportedly  been  attempts  to  use 
readily  available  troop  tallies,  known  as  “body  counts,”  to  assess  the  effect  of  the 
program.  It  is  argued  that  “body  counts”  are  perhaps  a  legitimate  MOP  as  they  likely 
speak  to  the  capacity  of  training  institutions,  but  they  do  not  alone  answer  end  state 
questions  as  to  how  the  program  has  made  a  difference. 

c.  The  Enduring  Need  for  Simplicity 

Michael  (2010)  also  observes,  “overly  detailed  assessment  and  collection  mires 
the  staff  in  the  creation  of  reports  and  briefs  to  the  detriment  of  performing  valuable 
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analysis  for  their  assessment”  (2010,  18)  It  is  useful  for  the  designer  of  the  assessment  to 
keep  the  end-user  in  mind;  especially  the  demands  that  any  proposed  assessment  would 
put  on  already  scarce  organizational  resources. 
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III.  GLOBAL  PEACE  OPERATIONS  INITIATIVE  AS  A  SYSTEM 


If  you  don ’t  understand  the  existing  system,  you  can ’t  be  sure  you  ’re 
rearchitecting  a  better  one. 

Susan  Ruth,  1993 

The  intent  of  this  chapter  is  twofold.  The  first  is  to  provide  enough  pertinent 
background  information  on  the  Global  Peace  Operations  Initiative  so  that  the  reader  has  a 
sufficient  degree  of  context  to  orient  herself  or  himself.  The  second  objective  is  to 
articulate  the  problem  space.  By  adequately  expressing  the  problem,  the  groundwork  will 
be  laid  for  the  solution  development.  Additionally,  a  well  understood  problem  is  an  ends 
unto  itself  as  it  will  often  produce  profitable  insights.  The  models  that  follow  in  chapters 
four  and  five  are  very  much  informed  by  how  the  problem  is  scoped,  bounded,  and 
defined  herein. 

While  the  GPOI  is  certainly  not  a  physics-  nor  a  software -based  system,  which  are 
perhaps  the  standard  subjects  of  application  of  SE  approaches,  the  GPOI  can  be  viewed, 
analyzed,  and  engineered  as  a  system.  Thinking  of  the  GPOI  as  a  system  is  principally  a 
translation  effort;  that  is  raw  GPOI  data  (e.g.,  GPOI  personnel  interviews,  policy 
documents)  must  be  rendered  into  system  engineering  artifacts  (e.g.,  stakeholder  analysis, 
functional  models,  problem  definition). 

A.  SYSTEM  CONTEXT 

What  follows  is  a  brief  summary  of  the  GPOI  with  emphasis  on  its  history, 
guiding  objectives,  and  program  management  structure.  Further  background  information 
is  readily  available  to  the  public,  notably  from  the  official  Department  of  State  website. 

1.  Brief  History  of  the  GPOI 

The  GPOI,  although  a  U.S.  government  managed  effort,  is  conducted  as  part  of  a 
larger  international  effort  to  increase  worldwide  peace  keeping  capacity.  According  to 
official  State  Department  documents,  the  GPOI  represents  the  United  States  component 
of  the  G8’s  “Action  Plan  for  Expanding  Global  Capability  for  Peace  Support  Operations” 
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as  developed  at  the  2004  G8  Sea  Island  Summit  (Bureau  of  Political-Military  Affairs  and 
Office  of  Plans,  Policy  and  Analysis  2013). 

The  initial  mandate  was  a  five-year  period  (2005-2009)  wherein  the  focus  was 
primarily  on  creating  capacity.  The  principal  end  state  was  to  create  a  large  number  of 
trained  personnel  that  could  be  employed  immediately  in  peacekeeping  operations, 
especially  on  the  African  continent  where  the  demand  signal  was  and  is  particularly  high. 
The  seven  high-level  objectives  for  the  first  phase  are  outlined  in  Appendix  D  the  most 
notable  of  which  is  the  goal  of  training  75,000  peacekeepers  for  deployment  in  Africa. 
According  to  the  State  Department,  this  goal  was  exceeded  by  the  end  of  phase  I  where 
by  their  count  nearly  87,000  peacekeepers  were  trained  (U.S  Department  of  State  n.d.) 

2.  GPOI  Today— Phase  II 

Today  the  GPOI  is  in  its  second  phase  of  operation  (2010-2014),  wherein  the 
focus  has  moved  to  supporting  partner  nations  in  developing  their  own  organic  capability 
to  train  and  equip  peacekeeping  forces.  The  overarching  theme  of  phase  II  is 
sustainability  (Bureau  of  Political-Military  Affairs  and  Office  of  Plans,  Policy  and 
Analysis  2013).  The  intent  is  for  GPOI  investments  to  be  made  to  have  lasting  impact  on 
a  country’s  ability  to  train,  maintain,  and  deploy  security  forces  for  regional  as  well  as 
international  United  Nations  security  missions.  The  phase  II  objectives,  as  outlined  in 
Table  1,  speak  to  this  modified  direction. 
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GPOI  PHASE  II  OBJECTIVES 

1.) 

In  coordination  with  other  U.S.  Government,  international  community,  and 
national  efforts,  assist  partner  countries  to  establish  and  strengthen  the  institutional 
infrastructure  required  to  achieve  and  sustain  self-sufficient  capability  to  conduct 

peace  operations  training; 

2.) 

Through  GPOTfacilitated  activities,  continue  to  train  peacekeepers  worldwide 
with  an  emphasis  on  train-the-trainer  instruction; 

3.) 

In  coordination  with  other  U.S.  Government  and  international  community  efforts, 
provide  support  to  deploying  units  to  address  partner  countries’  capacity  shortfalls; 

4.) 

Enhance  the  capacity  of  regional/sub-regional  organizations  and  institutions  to 
train  for,  plan,  deploy,  manage,  sustain,  and  obtain  and  integrate  lessons  learned 

from  peace  operations; 

5.) 

Enhance  efforts  to  establish  and  strengthen  the  institutional  infrastructure  and 
doctrinal  framework  required  to  train,  equip,  and  deploy  EPUs;  and 

6.) 

Support  the  continuation  and  enhancement  of  multilateral  approaches  and 
partnerships  to  coordinate  peace  operations  capacity  building  efforts. 

Table  1.  GPOI  Phase  II  Objectives  (after  Bureau  of  Political-Military  Affairs 
and  Office  of  Plans,  Policy  and  Analysis  2013,  2-4) 


It  is  anticipated  that  by  making  investments  in  infrastructure,  long-term 
relationship  building,  and  “train  the  trainer”  events  that  peace  keeping  capabilities  will 
become  institutionalized  in  the  partner  nations,  thus  providing  a  source  of  well  trained 
and  equipped  security  forces  for  years  to  come.  This  most  important  of  end  states,  which 
is  the  first  of  six  phase  objectives,  is  deemed  realized  when  a  country  has  reached  full 
training  capability  (FTC).  The  State  Department  has  a  defined  assessment  mechanism  for 
FTC,  which  will  be  discussed  in  later  chapters  (Bureau  of  Political-Military  Affairs  and 
Office  of  Plans,  Policy  and  Analysis  2013). 
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3.  Program  Management 

The  GPOI  is  managed  by  the  State  Department’s  Bureau  of  Politieal-Military 
Affairs  and  is  funded  through  the  peacekeeping  operations  (PKO)  account.  In  executing 
high  level  programmatic  functions  the  State  Department  works  closely  both  with  the 
Office  of  the  Secretary  of  Defense  (OSD)  and  the  Joint  Staff.  A  series  of  committees 
comprised  of  representatives  from  both  Department  of  State  (DOS)  and  Department  of 
Defense  (DOD)  provide  regional  and  strategic  direction  for  the  program.  As  shown  in 
Appendix  E,  the  appropriations  for  the  program  are  significant  at  levels  approaching 
SlOOmillion  a  year  (United  States  Department  of  State  n.d.). 

At  the  implementation  level,  a  collection  of  field  offices  carry  out  the  funded 
GPOI  activities.  According  to  the  State  Department,  “these  implementers  primarily 
include  DSCA  (Defense  Security  Cooperation  Agency),  the  COCOMs  (Combatant 
Commands),  DOS  regional  bureaus,  U.S.  diplomatic  posts,  and  contractors”  (Bureau  of 
Political-Military  Affairs  and  Office  of  Plans,  Policy  and  Analysis  2013). 

B.  PROBLEM  SPACE 

What  follows  is  an  attempt  to  decompose  the  problem  in  accordance  with 
standard  systems  engineering  principles.  The  first  task  is  to  look  at  the  stakeholder  set 
and  attempt  to  understand  their  needs  in  relation  to  one  another.  The  stakeholder  analysis 
is  intended  to  create  a  fundamental  requirement  statement  that  is  derived  from  the  needs 
of  the  customers  and  end  users.  A  functional  analysis  was  then  executed  to  decompose 
the  problem  space  and  better  understand  what  elements  will  be  required  in  the  solution 
space.  Finally,  the  problem  is  articulated,  but  not  before  the  assumptions,  limitations,  and 
scope  as  understood  are  stated. 

1.  Stakeholder  Analysis 

For  the  purposes  of  this  thesis,  a  stakeholder  is  anyone  who  has  an  interest  in  the 
problem  area  of  assessments  in  the  GPOI  and  its  solution.  In  the  GPOI  a  stakeholder  may 
take  on  many  different  roles  such  as  a  consumer  of  assessments,  a  producer  of 
assessments,  or  the  subject  matter  of  the  assessments  (Sage  and  Armstrong  2000). 
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The  key  elements  of  a  stakeholder  analysis  are  to  enumerate  the  stakeholders,  try 
to  quantify  their  objectives  and  values,  weigh  their  impact  on  the  system,  and  consider 
how  they  relate  to  one  another  (Trainor  and  Parnell  2011).  The  end  state  of  the  analysis 
would  be  deduced  system  requirements  as  well  as  potentially  identifying  problematic 
issues  to  be  considered  during  design. 

While  this  thesis  is  being  conducted  at  the  behest  of  USSOUTHCOM,  it  is 
important  to  consider  the  viewpoints  of  as  many  principal  stakeholders  as  possible. 
Dennis  Buede  (2000)  spoke  to  the  importance  of  considering  different  vantage  points 
when  he  explained. 

Each  stakeholder  has  a  significantly  different  perspective  of  the  system 
and  the  system’s  requirements.  If  one  perspective  is  single  out  as  the  only 
appropriate  one,  the  developers  of  the  system  will  miss  key  information, 
and  the  system  will  be  viewed  negatively  or  as  a  failure  from  the  other 
perspectives.  (2000,  122) 

a.  Stakeholder  Analysis  Methodology 

An  adapted  version  of  Manchester  Metropolitan  University’s  (MMU)  stakeholder 
analysis  toolkit  was  used  to  organize  the  stakeholder  analysis  (2014).  MMU’s  approach 
frames  stakeholder  analysis  as  a  component  of  project  management  and  thus  focuses  on 
practical  matters  that  pertain  to  achieving  some  defined  objective  set.  Specifically  the 
stakeholders  are  evaluated  in  two  general  ways:  the  first  is  by  what  each  entity  can 
contribute  to  the  system  and  second  is  the  obstacles  that  are  presented  by  not  getting  buy 
in  from  a  given  stakeholder.  This  world  view  is  deemed  applicable  to  the  GPOI 
assessments  problem.  Stakeholders  in  the  GPOI  assessments  arena,  in  addition  to  their 
need  for  information,  can  largely  be  defined  by  their  capacity  to  contribute  to  the 
assessment  system  and  how  the  system  is  limited  based  on  their  participation.  Since  the 
GPOI  as  a  system  is  composed  of  so  many  different  entities  (e.g.,  DOS,  DOD,  United 
Nations,  individual  sovereign  nations)  that  do  not  necessarily  have  to  cooperate  with 
another,  it  is  very  important  to  consider  how  their  participation  level  would  impact  a 
coordinated  assessment  framework.  (Manchester  Metropolitan  University  2014) 
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b.  First  Order  Stakeholder  Analysis 

A  system  as  large  and  complex  as  the  GPOI  will  possess  a  correspondingly  large 
set  of  stakeholders.  This  analysis  focuses  primarily  on  the  first  order  stakeholders.  A  first 
order  stakeholder  is  defined  as  an  entity  that  is  directly  producing,  consuming,  or  being 
the  subject  matter  of  the  assessment.  Thus  the  first  order  stakeholders  for  the  GPOI 
assessment  were  determined  to  be  the  implementer  set  (in  this  case  USSOUTHCOM), 
DOS,  the  participating  GPOI  nations,  and  the  United  Nations  Department  of 
Peacekeeping  Operations  (UN  DPKO).  There  are  of  course  several  other  important 
stakeholders,  but  if  they  could  be  understood  via  the  first  order  stakeholder  set,  they  were 
not  explicitly  enumerated.  For  example,  a  notable  admission  would  perhaps  be  the  United 
States  Congress.  While  Congress  does  fund  GPOI  (no  trivial  matter),  from  the  vantage 
point  of  assessments,  they  do  not  directly  contribute  to  the  production  of  assessments, 
and  while  they  do  consume  the  end  product,  it  is  through  the  intermediary  of  the  State 
Department. 

The  summarized  stakeholder  analysis  can  be  found  in  Appendix  F.  Of  course, 
there  is  a  qualitative  versus  quantitative  analysis  and  any  conclusions  drawn  are 
inherently  subjective  to  a  degree.  However,  the  exercise  does  contribute  to  the  overall 
analytic  effort  by  giving  a  measure  of  organization  to  what  is  a  fundamentally  ambiguous 
data  set. 


c.  Stakeholder  Analysis  Takeaways 

After  the  stakeholders  have  been  catalogued  in  a  rational  manner  (see  Appendix 
F),  the  intent  is  to  use  that  understanding  to  flesh  out  the  problem  space.  This  particular 
stakeholder  analysis  offered  some  important  requirements  as  well  as  a  few  notable  system 
characterizations . 

(1)  Traceability  as  a  Requirement.  All  of  the  consumers  require  traceability. 
Whether  it  is  the  State  Department  defending  the  program  or  a  COCOM  trying  to 
determine  why  an  activity  should  be  executed,  all  must  be  able  to  articulate  a 
fundamental  linkage  of  an  activity  executed  to  a  policy  objective  achieved. 
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(2)  Outcome  Measurement  as  a  Requirement.  All  of  the  consumers  want  the 
“So  what?”  question  answered.  Assuredly,  there  are  other  intermediary  questions,  such  as 
“how  did  this  training  event  go?”  or  “how  was  the  money  spent  on  this  building  project?” 
However,  all  of  the  stakeholders,  to  varying  degrees,  desire  to  know  outcomes  where 
impacted  (e.g.,  “how  has  this  line  of  effort  impacted  this  country’s  peacekeeping 
capacity?). 

(3)  State  Department  is  the  Principal  Stakeholder.  One  of  the  more  useful 
components  of  MMU’s  framework  is  that  it  encourages  the  SE  to  weigh  the  stakeholder 
set  by  their  impact  and  influence  on  the  problem  at  hand.  Impact  is  defined  as  how 
important  a  stakeholder’s  participation  is  to  the  success  of  the  system.  Influence  is 
defined  by  a  stakeholder’s  ability  to  move  the  system  in  some  direction  away  from  the 
status  quo.  For  example,  the  implementers’  set  is  defined  as  high  impact  (their 
participation  is  definitely  required)  but  medium  influence  as  they  in  large  part  are  subject 
to  overarching  policies  and  processes. 

The  DOS  as  a  stakeholder  is  assessed  as  high  impact  and  high  influence.  As  the 
administrators  of  the  program  of  record  (POR),  the  responsibility,  and  capacity,  for 
maintaining  and  modifying  the  strategic  vision  and  policy  rests  with  the  State 
Department.  If  a  different  function  of  the  GPOI  is  analyzed,  perhaps  the  conclusion  is 
different,  but  as  it  pertains  to  assessments  the  DOS  is  in  fact  the  principal  stakeholder. 
Certainly  the  DOD  is  included  and  contributes  strategic  thinking  capital,  but  if  a 
fundamental  shift  in  GPOI  assessments  is  to  be  accomplished,  it  is  unlikely  to  happen 
without  the  buy  in  from  DOS  leadership. 

(4)  Cooperation  and  Coordination  as  a  System  Limitation.  All  of  the  principal 
stakeholders  would  like  a  complete  system  view;  however,  none  of  them  on  their  own 
have  ready  access  to  the  information  that  would  enable  such  understanding.  The  degree  to 
which  data  can  be  aggregated  throughout  the  system  will  in  large  part  determine  the 
veracity  and  effectiveness  of  the  assessments.  Yes,  an  individual  GPOI  implementer  can 
undoubtedly  employ  a  better  assessment  model,  but  every  implementer  will  have  a  finite 
aperture  from  which  to  measure  the  system.  Stated  another  way,  the  stakeholder  set  has  a 
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fundamental  decision  to  make  when  pursuing  an  improved  assessment  framework:  either 
build  a  better  “stove  pipe”  or  pursue  a  more  cooperative  approach  across  agency 
boundaries.  This  thesis  of  course  argues  for  the  latter  approach. 

2.  Functional  Analysis 

Functional  analysis,  as  described  by  Trainor  and  Parnell  (2011),  is  “a  systematic 
process  to  identify  the  system  functions  and  interfaces  required  to  achieve  the  system 
objectives”  (2011,  315).  And  a  Function  is  defined  as  “a  characteristic  task,  action,  or 
activity  that  must  be  performed  to  achieve  a  desired  outcome”  (Trainor  and  Parnell  2011, 
315).  If  a  system’s  function  and  associated  interactions  are  not  well  understood,  it  is 
difficult  to  develop  a  cogent  requirement  set,  and  without  a  proper  requirement  set,  it  is 
less  likely  that  the  developed  solution  will  answer  the  stated  problem. 

While  this  thesis  is  narrowly  focused  on  the  assessment  function  of  the  GPOI 
system,  assessments  cannot  be  studied  in  isolation  because  within  a  system  functions 
fundamentally  interact  with  each  other.  Thus,  the  end  state  of  this  functional  analysis  is  to 
decompose  the  GPOI  system  only  as  far  as  necessary  to  orient  the  assessment  function 
within  the  broader  system.  By  understanding  how  the  assessment  function  interacts  with 
the  other  functions  the  connections  can  be  characterized  and  defined.  These  connections 
(i.e.,  inputs,  outputs)  will  form  the  basic  requirement  set  for  an  assessment  function  that 
will  behave  rationally. 

What  follows  is  an  overview  of  how  the  functional  analysis  was  developed  and 
the  understanding  that  was  gained  in  the  process. 

a.  Functional  Hierarchy 

Functional  analysis  is  in  large  part  a  decomposition  exercise.  What  first  must  be 
decided  is  the  top  line  function  to  be  explored.  In  this  case,  the  overarching  function  of 
“Administer  the  GPOI”  function  was  chosen.  The  “Administer  the  GPOI”  function  is 
defined  here  as  the  sum  total  of  activities  that  must  be  accomplished  by  the  U.S. 
government,  in  concert  with  international  partners,  to  achieve  the  policy  objectives  of  the 
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initiative.  The  idea  is  to  choose  a  level  of  abstraction  high  enough  to  ensure  that  the 
function  of  interest  is  captured,  which  in  this  case  is  the  “assessments  function”  (Trainor 
and  Parnell  2011). 

Once  the  high  level  function  was  chosen,  “Administer  the  GPOI,”  all  of 
the  supporting  functions  were  then  enumerated.  The  functions  are  then  grouped  and 
ordered  logically,  which  produces  a  functional  hierarchy.  Of  course,  “Assess  Progress” 
was  chosen  as  a  first  order  function  intentionally  so  as  to  get  a  view  of  the  system 
from  a  specific  vantage  point.  Figure  5  is  a  graphical  illustration  of  the  proposed  high 
level  GPOI  Functional  Hierarchy.  A  more  detailed  functional  hierarchy  is  offered  in 
Appendix  G. 


Figure  5.  High  Level  GPOI  Functional  Hierarchy 


The  other  three  first  order  functions  are:  “manage  the  program,”  “obtain  funding,” 
and  “implement  GPOI.”  The  “manage  the  program”  function  is  the  high  level  program 
management  function  that  encompasses  activities  such  as  developing  processes  and 
procedures  and  providing  oversight.  The  “obtain  funding”  function  is  concerned  with  all 
of  the  activities  necessary  to  request,  appropriate,  and  disperse  monies  necessary  to  fund 
GPOI  activities.  The  “Implement  GPOI”  function  is  comprised  of  all  of  the  activities  that 
are  necessary  to  realize  GPOI  events. 
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b.  Functional  Modeling 

The  next  step  is  to  characterize  how  the  functions  interact  with  each  other.  As  an 
intermediary  step  a  data  flow  diagram  (DFD)  was  constructed  to  help  consider  the  nature 
of  these  relationships  (see  Appendix  H).  A  DFD  is  a  representation  of  the  transmission  of 
data  within  information -based  systems  (Buede  2000).  The  functional  model  is  then  more 
clearly  and  completely  expressed  using  a  formal  modeling  language.  In  this  case,  IDEFO 
was  chosen  for  its  ability  to  clearly  express  interactions  of  organizational  activities  as  a 
process.  For  a  brief  overview  of  the  chosen  modeling  language,  see  Appendix  I  (Software 
Engineering  Standards  Committee  of  the  IEEE  Computer  Society  1998). 

The  proposed  GPOI  functional  model  (see  Eigure  6)  expresses  how  the 
assessment  relates  to  the  rest  of  the  system.  The  nature  of  the  connections  was  developed 
from  interviews  with  GPOI  personnel  and  policy  as  described  in  the  GPOI 
Implementation  Guide  (GIG)  (Bureau  of  Political-Military  Affairs  and  Office  of  Plans, 
Policy  and  Analysis  2013).  In  this  case,  the  importance  of  using  codified  policy 
documents  to  construct  the  functional  model  is  to  express  the  system  “as  it  is”  or  “as  it 
should  be.”  The  idea  is  to  remake  the  assessment  for  the  GPOI  and  not  the  other  way 
around.  Hence,  the  rest  of  the  GPOI  system  (e.g.,  organizational  structure,  funding 
processes,  foundational  national  policies)  are  by  definition  design  constraints. 
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**GRC  -  GPOl  Regional  Committee 

NODE:  |tITLE:  GPOI  Functional  Modeling  |nO.: 

Figure  6.  GPOI  Functional  Model  in  IDEFO 

The  IDEFO  model  (Figure  6)  views  the  GPOI  as  a  process  by  expressing  the 
nature  of  the  connections  between  the  various  first  order  functions  as  previously 
enumerated  (see  Figure  5). 

c.  Application  and  Key  Takeaways  from  the  Functional  Analysis 

Functional  analysis  is  both  an  end  and  a  means.  It  is  an  end  unto  itself  as  it  a 
process  model  for  the  GPOI  system,  that  if  expressed  correctly,  properly  conveys  intra¬ 
system  interaction.  As  a  means,  the  functional  analysis  lays  the  groundwork  for  a 
properly  articulated  problem  statement.  See  Appendix  J  for  further  discussion  on  the 
application  and  salient  observations  of  the  functional  analysis. 
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3. 


Problem  Articulation 


The  stakeholder  and  functional  analysis  provided  a  means  of  thinking  about  and 
organizing  an  initial  requirement  set.  What  is  needed  next  is  to  properly  bound  the 
problem  space  and  apply  caveats  as  understood.  Thus,  a  working  problem  statement  is 
developed. 


a.  Assumptions 

It  is  assumed  that  discretionary  budget  that  can  be  applied  to  any  proposed 
improvements  of  assessments  will  be  minimal.  While  the  case  for  additional  budget 
towards  improved  assessments  will  be  made  throughout  the  research,  the  proposed 
assessment  products  must  attempt,  to  the  extent  possible,  to  use  existing  resources  or 
offer  some  efficiency  in  exchange. 

As  a  corollary  to  the  understood  budgetary  constraints,  it  is  assumed  that 
“organizational  will”  is  not  a  limiting  factor.  For  any  assessment  product  or  process  to  be 
successfully  actualized,  it  must  have  the  buy  in  from  leadership  to  the  point  where  it  can 
be  institutionalized.  While  the  solution  space  is  viewed  as  fundamentally  limited  by 
material  resource  application,  it  will  not  be  viewed  as  limited  by  immaterial  resource 
application.  Material  resources  are  defined  as  those  that  can  be  readily  monetized  such  as 
budgets  or  staff  labor  hours.  Immaterial  resources  are  those  means,  while  not  necessarily 
possessing  directly  measureable  monetary  value,  are  nonetheless  valuable  and  necessary 
to  carry  out  objectives.  An  example  of  an  immaterial  resource,  as  it  pertains  to  this 
discussion,  would  be  inter-agency  cooperation  or  intra-organizational  consensus  building 
(e.g.,  an  organization’s  intrinsic  ability  to  adopt  and  actualize  a  product).  These 
immaterial  resources  are  by  no  means  inconsequential;  in  fact,  they  may  prove  to  be  the 
greatest  impediment  to  achieving  an  improved  assessment  framework.  There  are  assumed 
to  not  be  limitations  because  if  any  suggested  product  is  adopted,  limitations  will  have 
been  effectively  obviated.  Which  is  only  to  say  that  the  research  attempts  to  address 
organizational  issues  that  impact  the  assessment,  but  the  burden  for  ultimately  reconciling 
organizational  issues  will  rest  with  the  program  offices  and  COCOMs  in  question. 
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b.  Scope 

The  idea  of  scoping  is  to  make  a  intentional  decisions  as  to  the  boundaries  of  what 
the  problem  encompasses s.  The  intent  is  to  focus  the  effort  and  what  is  central  to  the 
stakeholder’s  need,  as  well  as  achievable  by  the  researcher,  and  exclude  all  is  well.  As 
Maier  and  Rechtin  (2009)  put  it,  “Desirably,  scoping  limits  what  needs  to  be  considered 
and  why”  (2009,  259). 

It  is  held  that  the  endstate,  or  the  value,  of  any  assessment  is  its  contribution  to  the 
decision  maker’s  capacity  to  make  sound  course  corrections  for  his  or  her  program  or 
campaign.  The  decision-makers  are  the  State  Department  GPOI  program  managers  who 
are  responsible  for  making  sound  investments  and  the  COCOM  commanders  that  must 
incorporate  the  GPOI  within  a  larger  theatre  of  operations.  Thus,  the  objective  of  the 
assessment  should  be  actualized  by  the  arc  of  observed  data  to  program  feedback. 

This  arc,  as  it  pertains  to  the  GPOI,  is  represented  in  Figure  7  as  a  continuous 
measurement  and  feedback  cycle.  The  assessment  function  can  be  decomposed  into  into 
three  sub-functions:  data  collection  function,  analysis  function,  and  decision-making 
function.  The  data  collection  function  is  an  umbrella  capability  that  encompasses  all  the 
activities  that  pertain  to  system  measurement  and  data  aggregation.  The  analysis  function 
is  defined  by  the  capacity  to  make  both  descriptive  and  inferential  conclusions  of  a  given 
data  set  and  to  generate  logical  recommendations.  Finally,  the  decision-making  function 
represents  the  capacity  to  process  analytic  products  in  a  larger  context  and  execute 
decisions  (system  feedback).  While  the  end  state  of  the  work  is  to  ultimately  improve  the 
decisionmaking  function,  it  has  been  determined  to  focus  the  prinicipal  research  effort  on 
the  data  collection  function. 
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Figure  7.  Assessment  Cycle 


The  reasons  for  focusing  on  the  data  collection  are  twofold.  The  first  is  a  practical 
matter  of  organizational  mechanics.  That  is,  the  GPOI  program,  to  included  elements 
from  both  State  and  Defense,  have  well  developed  and  codified  decision  making 
apparatuses.  Additionally,  while  analytic  resources  may  not  be  adequately  matrixed,  the 
GPOI  program  offices  and  COCOMS  both  possess  organic  analytic  capability  in  the  form 
of  organized  staffs.  However,  the  data  collection  function  appears  to  be  the  least  defined 
function  and  likely  lacks  both  resources  and  a  cogent  organizational  framework. 

The  second  reason  for  focusing  on  the  data  collectionis  that  it  is  suspected  that  the 
necessary  data  is  not  now  being  collected.  The  best  analysts  cannot  be  reasonably 
expected  to  exceed  the  quality  of  their  available  data  and  the  soundness  of  a  program 
manager’s  decision  perhaps  will  not  exceed  the  quality  of  their  conclusions.  Having  an 
adequate  data  set  is  necessary  for  a  functional  decision  process. 


c.  Problem  Summary 

A  system  is  successful  when  a  natural  intersection  of  technology, 
politics,  and  economics  is  found. 

A.  D.  Wheelon  1986 

The  difficulty  in  assessing  the  GPOI  can  be  characterized  by  two  fundamental 
problem  areas.  The  first  is  a  lack  of  an  intra/inter  agency  agreement  and  organizational 
resource  management  being  directed  at  the  assessment  space.  The  second  is  the 
incomplete  construction  of  the  assessment  models.  The  first  problem  is  a  principally 
social  and  the  second  is  principally  technical,  but  they  both  can  be  addressed.  It  is 
doubtful  that  addressing  one  and  not  the  other  will  offer  any  substantive  gains. 

(1)  Lack  of  an  Explicit  Framework.  As  alluded  to  earlier,  flawed  assessment 
models  are  themselves  symptomatic  of  a  larger  problem.  Imagine  an  individual,  he  or  she 
could  be  a  contractor,  a  military  foreign  officer  or  an  embassy  employee,  in  country  with 
an  assessment  model  in  hand.  That  person  may  be  extraordinarily  perceptive  and  have  the 
perfect  set  of  metrics  in  hand,  but  if  her  analysis  is  not  part  of  a  coordinated  effort,  then 
ultimately  it  may  not  be  of  any  consequence. 

What  is  lacking  is  an  overarching  framework  that  informs  the  various  components 
of  the  GPOI  assessment  system  what  they  need  to  measure.  What  is  missing  is  someone 
or  some  organization  with  a  sufficiently  broad  view  to  “architect”  a  system  that 
coordinates  the  measurement  efforts  across  the  enterprise. 

This  framework  needs  to  be  codified  if  it  is  to  be  institutionalized.  Additionally, 
what  is  missing  from  the  framework  is  an  overall  assessments  hierarchy.  Processes, 
procedures,  and  applicable  doctrine  must  also  be  made  explicit.  In  the  USSOUTHCOM 
process  handbook,  there  are  two  codified  processes  for  GPOI.  One  details  how  a  FTC 
assessment  is  routed  for  approval,  and  the  other  explains  the  budget  cycle  as  it  pertains  to 
funding  individual  activities.  These  two  processes  are  not  necessarily  faulty,  but  much 
more  must  be  mapped  out  in  order  to  have  a  robust  assessments  program  (United  States 
Southern  Command  Process  Management  and  Analysis  Cell  2013). 

(2)  Missing  and  Fimited  Measurement  Models.  As  far  as  this  researcher  is 


aware,  there  is  only  one  assessment  model  in  circulation.  This  would  be  the  Full  Training 
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Capability  (FTC)  assessment.  The  second  phase  of  GPOI  has  six  clearly  stated  objectives, 
but  it  appears  that  the  FTC  only  maps  to  the  first  objective  (see  Table  1).  How  are  the 
other  five  objectives  being  measured? 

The  FTC  does  in  fact  map  back  to  the  first  state  GPOI  objective  of  phase  IF  But 
in  determining  achievement  of  that  objective,  or  any  of  the  others  for  that  matter,  is 
enough  insight  provided  to  answer  the  “so  what”  question?  It  speaks  to  how  the  activities 
have  mapped  to  a  stated  program  objective,  but  the  broader  arc  of  policy  objective  to 
outcome  is  conspicuously  absent. 

For  example,  imagine  that  objective  one  (FTC)  and  objective  three  (support 
provided  for  deploying  units)  are  verified  to  have  been  met  for  a  particular  country.  Does 
anyone  know  if  it  meant  anything?  That  is  did  those  troops  deploy  and  how  did  they  do? 
Was  the  mission  successful? 

The  DOS  seems  to  concur  in  part.  The  official  DOS  GPOI  website  they  state, 
“The  program  has  a  substantial  metrics  and  evaluation  component  which  is  guided  by  the 
following  outcome-oriented  considerations:  actual  deployments,  effectiveness  in  PSOs, 
improvement  of  capacities,  and  self-sufficiency”  (United  States  Department  of  State 
n.d.).  If  outcomes  are  being  properly  measured  as  parts  of  a  coordinated  assessment 
process,  this  researcher  is  unaware  of  it. 
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IV.  ARCHITECTING  A  GPOI  ASSESSMENTS  FRAMEWORK 


The  intent  of  this  chapter  is  threefold.  First,  the  necessities  and  the  uses  of 
coherent  systems  architecture  are  outlined.  Secondly,  a  working  enterprise  level 
architecture  for  assessing  the  GPOI  is  offered  with  an  accompanying  explanation  as  to 
how  it  was  developed  and  how  it  should  be  interpreted.  And  finally,  a  discussion  on  how 
the  proposed  architecture  can  be  adopted,  employed  to  good  effect,  and  further  improved. 

A.  ARCHITECTURE  DEVELOPMENT  IN  THE  GPOI 

There  are  two  realities  that  have  made  systems  architecting  necessary  to  address 
the  challenge  of  assessing  the  GPOI.  The  first  is  that  the  problem  is  so  large,  so  complex, 
and  so  ill-defined  that  it  requires  a  degree  of  structure  to  organize  its  disparate  elements. 
Secondly,  this  problem  presents  the  need  to  synthesize  across  distinct  domains;  there  are 
perhaps  several  ways  to  adequately  address  the  problem.  However,  any  suitable  solution 
would  certainly  have  to  incorporate  both  technical  as  well  as  organizational  elements. 
Both  the  technical  and  organizational  mechanisms  will  need  a  degree  of  coherency  if  the 
assessment  system  is  to  succeed. 

I.  Characteristics  of  a  Sound  Architecture  for  the  GPOI 

This  section  is  both  a  primer  on  how  to  think  about  architectures  as  well  as  a 
discussion  on  what  are  believed  to  be  the  important  characteristics  of  operational 
assessments  architectures  for  application  in  the  Global  Peace  Operations  Initiative. 

a.  Structure  and  Completeness 

Defining  an  operational  assessments  architecture  in  the  Global  Peace  Operations 
is  a  complex  problem  to  understand  and  therefore  a  difficult  problem  to  define.  What  is 
required  is  a  definitive  means  of  assessing  the  system,  which  is  inherently  technical  and 
complex  in  nature,  through  the  introduction  of  a  degree  of  structure  via  systems 
architecting.  This  gives  both  a  starting  point  for  specific  design  as  well  as  offering 
traceability  back  to  problem  set. 
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b.  Holding  the  Technical  in  Balance  with  the  Non-Technical 

The  various  elements  of  the  GPOI  assessment  system  (for  both  the  current  and 
proposed  versions)  can  be  binned  into  two  categories:  technical  and  organizational. 
Technical  elements  would  include  the  assessment  mechanisms  themselves  (that  is  a 
defined  set  of  metrics  that  organizes  system  measurement).  The  organizational  elements 
are  those  that  include  the  very  important  agreements,  processes,  and  procedures  that 
enable  assessment  to  take  place  and  synthesized  in  a  meaningful  way.  Since  GPOI 
assessments  requires  cooperation  from  both  inter  and  intra  organizational  elements,  this 
facet  of  the  problem  space  takes  on  added  importance. 

c.  A  Way  to  Think  about  Architectures 

Before  presenting  the  development  of  the  GPOI  assessments  architecture,  a 
general  discussion  on  how  to  think  about  architectures  is  offered.  The  intent  is  for  the 
reader  to  gain  a  degree  of  context  as  to  where  architectures  can  be  used  in  the  design 
process. 

One  way  to  think  about  architectures  is  a  series  of  viewpoints.  Looking  at  a 
system  from  different  angles  and  at  different  levels  of  abstraction  can  help  the  designer 
consider  the  problem  in  greater  totality.  Depending  on  the  framework  employed,  a 
different  part  of  the  system  can  be  understood.  In  their  paper,  “Enterprise  Architecture 
Tools  for  Delivering  Combat  Capability,”  authors  Steve  Carey  and  Michael  Jacobs 
present  three  views  for  a  given  system:  operational  view,  systems  view,  and  technical 
view  (n.d.).  The  operational  view  focuses  on  characterizing  the  interaction  of  the  high 
level  tasks  (e.g.,  what  people  have  to  accomplish  within  the  system  that  is  important  to 
the  operation).  The  systems  view  highlights  how  the  constituent  systems,  or  subsystems, 
interact  to  achieve  a  set  of  capabilities.  The  technical  view  brings  the  architecture  down 
to  a  set  of  rules  or  requirements  that  inform  the  actualizing  of  the  various  systems  and 
subsystems.  These  three  views  are  expressed  graphically  in  Figure  8. 
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Figure  8.  Levels  of  Architecture  Frameworks  Explained  (from  Carey  and 

Jacobs  n.d.) 


In  this  thesis,  an  operational  view  was  used  as  starting  point  to  begin  thinking 
about  the  important  elements  and  how  they  interacted.  For  development  purposes,  an 
operational  concept  was  developed,  see  Appendix  K.  The  architecture  as  proposed  in  this 
chapter  can  be  characterized  as  “system  views”  that  focus  on  articulating  interactions  of 
the  major  systems  to  achieve  suitable  assessment  of  the  GPOI.  The  final  step  in  the 
architecting  process  is  to  drive  to  the  “technical  views,”  which  are  defined  as  the  required 
elements  to  execute  the  architecture.  These  actualizing  elements  in  support  of  the 
proposed  architecture  are  discussed  in  Chapter  V. 
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2. 


Architectural  Coherency:  Business  and  Systems  Alignment 


There  is  no  such  thing  as  a  purely  technical  solution 

Brenda  Foreman  1990 


a.  Choosing  an  Architecting  Paradigm 

When  selecting  a  framework  to  give  coherency  to  the  proposed  architecture,  two 
characteristics  were  sought:  traceable  capability  management  and  the  ability  to  balance 
technical  and  organizational  architectures.  The  architectural  paradigm  chosen  for 
application  to  the  GPOI  assessments  problem  was  a  model  developed  by  Totem  Ltd. 
Totem  is  a  New  Zealand  IT  firm  that  proposes  using  a  cascading  series  of  architectures  to 
translate  desired  capabilities  into  application  architectures  that  will  then  define  the 
required  systems.  Totem’s  architecting  paradigm  is  represented  in  Figure  9  (Totem  Ltd. 
2011). 
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Figure  9.  Totem  Ltd.’s  Model  for  the  Alignment  of  Business  Capability  and  IT 

Architecture  (from  Totem  Ltd.  2011) 


Totem’s  architecture  is  based,  in  part  at  least,  on  capability  management  as 
commonly  used  in  the  DOD  and  the  United  Kingdom’s  Ministry  of  Defence  (MOD).  The 
idea  of  capability  management,  central  to  the  systems  design  process,  is  to  focus  on 
defining  the  needs  first  and  then  examining  the  array  of  means  to  address  them.  By 
focusing  on  capability,  the  aperture  of  design  is  opened  up  to  other  opportunities  outside 
of  the  pre-conceived  solution  set.  In  the  DOD  world,  this  spectrum  of  solutions  is 
commonly  referred  to  as  doctrine,  organizations,  training,  materiel,  leadership 
development,  personnel,  and  facilities  (DOTMLPF)  (Defense  Acquisition  University 
2014).  The  implication  of  capability  management  and  the  idea  of  thinking  about  solution 
spaces  (like  DOTMLPF)  is  that  even  if  the  right  solution  is  that  material  item,  there  are 
likely  other  elements  that  are  necessary  to  actualize  it  (e.g.,  codified  doctrine  and  the 
personnel  component).  This  application  of  capability  management  described  above  is 
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important  not  only  because  it  addresses  traceability  but  also  includes  other  elements  of 
the  solution  space  beyond  technical  mechanisms.  Thus,  in  the  proposed  GPOI 
assessments  model  described  in  the  following  section,  the  top  line  framework  is  a 
capabilities-based  architecture. 

To  realize  the  aforementioned  capabilities  architecture.  Totem  proposes  an 
overarching  functional  architecture  that  translates  the  capabilities  enumerated  as 
requirements  into  specific  functions.  The  idea  is  not  just  to  enumerate  and  decompose  the 
necessary  functions  but  to  develop  their  interactions  with  other  functions  and  system 
elements  until  it  is  made  explicit  as  to  what  each  function  needs  to  achieve  (Totem  Ltd. 
2011). 

The  functional  architecture  is  then  bifurcated  into  application  architecture  and  an 
organizational  architecture.  This  emphasis  on  explicitly  addressing  the  technical  and 
organizational  elements  as  they  support  the  functional  architecture  was  applicable  to 
GPOI  for  the  aforementioned  reason  of  balancing  the  right  technical  products  with  the 
necessary  organizational  support  (Totem  Ltd.  2011). 

b.  Adapting  the  Architectural  Framework  for  the  GPOI 

Totem’s  original  model,  as  described  above  and  depicted  in  Figure  9  is  adapted 
and  expanded  for  the  application  of  the  GPOI  assessment  problem.  In  the  proposed 
architectural  framework,  seen  in  Figure  10,  three  “spaces”  were  defined:  structured 
problem  space,  structured  solution  space,  and  actualizing  element  space. 
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Figure  10.  Proposed  Assessments  Framework  (after  Totem  Ltd.  201 1) 


The  highest  level  is  labeled  “structured  problem  space,”  which  includes  the 
capabilities-based  architecture.  This  capabilities-based  architecture  describes  the  systems 
in  terms  of  what  it  should  be  required  to  do.  The  capabilities  architecture  was  developed 
from  the  problem  definition  phase  of  the  research  (see  Chapter  III). 

The  next  layer  in  the  framework  is  the  “structured  solution  pace,”  which  includes 
the  functional  architecture,  which  in  turn  is  decomposed  into  a  data-based  architecture 
and  an  organizational  architecture.  This  “structured  problem  space”  is  primarily  a  set  of 
systems  views  that  focus  on  the  defining  interactions  of  the  necessary  constituent  systems 
that  would  make  up  a  suitable  assessments  framework. 

The  third  and  final  layer  is  the  “actualizing  element  space”  which  is  principally  a 
technical  view.  Actualizing  elements  are  defined  as  those  components  that  support  the 
various  subsystems  in  achieving  system  level  capabilities.  Moving  from  the  structured 
solution  space  to  the  actualizing  element  space  represents  the  translation  of  what  is  an 


49 


abstraction  of  the  system  to  real  world  products  that  enable  assessments  to  be  executed. 
The  actualizing  elements  are  discussed  in  greater  detail  in  Chapter  V. 

3.  Capabilities-Based  Architecture 

The  capabilities  architecture,  describes  the  system  at  the  level  of  abstraction  in 
terms  of  “what  it  needs  to  do”  to  achieve  its  system  objective.  What  follows  is  a 
discussion  on  the  development  and  interpretation  of  the  capabilities  architecture  as  well 
as  its  implications. 

a.  Proposed  Capabilities-Based  Architecture 

The  chosen  system  objective  for  assessments  in  the  GPOI  is  simply  “understand 
the  GPOI.”  This  one  umbrella  objective  effectively  encompasses  the  stakeholder  set’s 
overall  needs.  By  enumerating  the  capabilities  necessary  to  “understand  the  GPOI,”  a 
complete  and  effective  assessments  architecture  can  be  developed.  Figure  11  represents 
the  proposed  capabilities  architecture  for  assessing  the  GPOI. 

The  overarching  capability  of  “Understand  the  GPOI”  was  decomposed  into  four 
supporting  capabilities:  (1)  Understand  GPOI  policy  and  objectives  (2)  Understand  cost 
of  conducting  the  GPOI  (3)  Understand  how  GPOI  is  actualized  and  (4)  Understand 
GPOFs  impact.  As  discussed  in  the  theory  section  of  Chapter  II,  the  intent  is  to 
decompose  in  such  a  way  so  as  to  encompass  all  of  the  higher  level  element  as  well  as 
reducing,  if  not  eliminating,  overlap  between  elements  of  the  same  level.  When 
explaining  the  four  main  elements  of  system  understanding,  understanding  comes  from 
assessment.  For  the  purposes  of  this  thesis,  assessment  is  defined  as  the  formal 
measurement  of  a  system  (or  a  part  of  a  system)  by  using  a  defined  metric.  Thus,  there 
are  two  parts  of  assessments:  understanding  what  needs  to  measured  and  why  and  having 
the  capacity  and  access  to  measure. 
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Figure  1 1 .  GPOI  Capabilities-Based  Architecture 

As  an  aside,  the  capacity  to  access  and  to  measure  a  system  is  certainly  not  trivial. 
As  historical  assessments  in  the  recent  conflicts  in  Iraq  and  Afghanistan  have  shown, 
there  is  a  great  temptation  for  assessors  to  measure  the  portions  of  the  system  that  are 
easily  measured  that  may  not  represent  the  required  data  set  (Michael  2010).  There  needs 
to  be  coherency  between  the  metrics  (or  assessment  models)  that  are  properly  developed 
and  measurement  mechanisms  that  contributes  to  those  models. 

(1)  Understanding  GPOFs  Policy  and  Objectives.  The  capability  “Understand 
GPOI  Policy  &  Objectives”  does  not  directly  address  assessing  the  system  itself.  Rather  it 
acts  as  a  feedback  loop  to  help  ensure  that  the  basis  and  mechanisms  of  the  assessment 
framework  are  sound.  The  capability  can  be  better  understood  by  further  decomposing  it 
into  the  two  sub-capabilities  of  “Assess  GPOI  Policy  &  Objectives”  and  “Assess  GPOI 
Processes  and  Procedures.”  This  essentially  addresses  whether  the  GPOI  program 
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management  has  formulated  the  correct  objectives  and  policy  goals  and  whether  the 
assessment  activity  has  formulated  the  proper  processes  and  procedures  in  response  to  the 
strategic  direction. 

The  assessment  framework  is  expected  to  be  executed  at  several  levels  below  the 
policy  formation  level.  However,  the  staff  officers  (military,  civilian,  and  contractors) 
tasked  with  enacting  the  assessment  framework  should  be  able  to  look  outside  their 
operating  level  and  ask  two  questions:  (1)  Are  the  doctrine,  processes,  and  procedures 
that  are  used  to  conduct  the  assessments  in  line  with  the  objectives  and  policies  that  have 
been  handed  down?  (2)  Are  the  objectives  and  policy  (GPOI)  consistent  with  the  highest 
level  policy  (national)  and  overarching  strategy?  As  will  be  shown  later  in  this  chapter, 
this  is  not  trivial  as  the  overarching  objectives  for  the  GPOI  have  not  been  sufficiently 
articulated  to  allow  for  proper  assessment. 

(2)  Understanding  the  Cost  of  Conducting  the  GPOI.  The  intent  of  this 
capability  is  to  capture  the  important  inputs  of  the  GPOI  system.  Stakeholders,  notably 
Congress  and  the  State  Department,  are  concerned  about  the  arc  of  money  to  real 
outcomes.  Besides  the  significant  financial  outlays  there  are  other  inputs  such  as 
organizational  resources  (human  capital  or  facilities).  This  “other”  category  could  either 
be  expressed  as  opportunity  costs  or  they  can  themselves  be  monetized  and  rolled  into  the 
total  financial  expenditure.  The  important  point  to  be  made  here  is  that  while  this 
accounting  function  is  believed  to  be  executed  diligently,  it  must  be  incorporated  into  the 
assessments  process  at  some  point.  As  will  be  discussed  later,  one  of  the  goals  is  to  assess 
correlations  and  causation  of  system  inputs  to  system  outcomes.  As  the  architecture  is 
developed  into  assessment  models  these  “costs”  are  characterized  as  system  inputs  and 
their  assessments  are  referred  to  as  measures  of  input  (MOIs). 

(3)  Understanding  How  GPOI  is  Actualized.  This  capability  speaks  to  the 
need  to  assess  how  a  given  GPOI  event  was  executed.  This  is  better  understood  by 
furthering  decomposing  into  two  sub-capabilities:  “Assessing  GPOI  Events”  and 
“Assessing  GPOI  Objectives.”  These  capabilities  answer  the  questions:  Were  the  GPOI 
events  conducted  properly?  And  did  the  events  achieve  their  intended  objective(s)?  These 
are  separate  questions  and  therefore  separate  assessments.  It  is  possible  to  conduct  a 
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GPOI  event  in  accordance  with  procedure  and  policy  and  not  achieve  the  objective  and  it 
is  possible,  although  less  likely,  to  conduct  a  GPOI  event  outside  of  doctrine  and  still 
achieve  an  objective.  The  first  question  (processes)  gives  the  GPOI  PM  insight  into  how 
well,  and  how  precisely,  the  program  is  being  executed.  The  second  question  gives  the 
GPOI  program  manager  insight  into  how  the  given  event  is  contributing  to  the  stated 
objectives. 

The  “Assess  GPOI  Events”  speaks  to  the  procedural  compliance  of  the  particular 
event.  Basically,  was  the  event  (e.g.,  build  the  school,  train  the  trainers,  give  material  aid) 
done  in  accordance  with  statute,  process  and  procedure?  This  is  important  both  for 
government  accountability  measures  (expenditure  of  resources)  as  well  as  for  application 
of  codified  doctrine.  If  the  procedures  are  insufficient,  then  the  implementers  should 
suggest  adoption  of  “best  practices”  as  they  have  learned  in  the  field.  This  capability  is 
manifested  later  in  the  assessment  models  as  a  measure  of  processes  (MOPr)  (Santos 
2011). 

The  “Assess  the  GPOI  Objectives,”  capability  speaks  to  being  able  to  understand 
what  the  GPOI  is  achieving.  These  are  principally  immediate  results  and  not  end  states. 
As  mentioned  in  Chapter  III  the  GPOI  has  a  clear  set  of  objectives  against  which  can  and 
should  be  measured.  This  capability  is  manifested  later  in  the  assessment  models  as  a 
measure  of  performance  (MOP). 

(4)  Understand  GPOFs  Impact.  And  finally  the  “Understand  GPOFs  Impact” 
is  the  capability  to  assess  whether  the  efforts  of  the  GPOI  mattered.  The  assessment 
framework  should  measure  the  system  against  a  set  of  “so  what?”  metrics.  This  is 
achieved  by  making  explicit  what  the  desired  outcomes  or  end  states  of  the  program  are. 
This  is  a  fundamentally  different  capability  then  being  able  to  determine  if  the  GPOI 
event  achieved  an  objective.  For  example  an  event  could  achieve  the  stated  objective  to 
provide  support  for  deploying  units  (see  Phase  II  objectives  in  Table  1)  but  may  or  not 
have  contributed  to  a  desired  outcome. 
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b.  Implications  of  the  Capability  Architecture 

A  capabilities-based  architecture  offers  two  principal  uses.  The  first  is  that  it 
describes  capabilities  such  that  a  sound  functional  architecture  can  be  constructed,  as 
developed  in  the  next  section.  Secondly,  it  offers  a  means  of  critiquing  the  current 
assessments  framework  by  highlighting  potential  gaps  in  capabilities.  Figure  12  is  a 
visual  representation  of  the  discussion,  which  is  discussed  in  more  detail  in  the  following 
sections. 


Assessment  Process. 

Figure  12.  Proposed  Capabilities  Architecture  against  Existing  GPOI 

Assessment  System 


(1)  What  is  Missing  from  Understanding  GPOFs  Policy  and  Objectives?  As 

far  as  can  be  discerned  this  feedback  loop  is  not  a  current  part  of  the  formal  assessment 

process,  at  least  at  the  implementer  level  as  studied.  Elements  of  program  review  no 

doubt  occur  at  different  levels  throughout  the  GPOI  organization;  however,  it  is  a 
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symptom  of  not  making  the  feedback  mechanism  explicit,  which  makes  the  assessments 
framework  incomplete.  These  potential  shortcomings  are  best  explained  by  examining 
the  other  three  sub-capabilities. 

(2)  What  is  Missing  from  an  Understanding  the  GPOFs  Cost?  The  accounting 
element  of  GPOI  is  likely  conducted  properly  and  completely.  However,  it  is  not  clear  if 
this  information,  which  is  likely  resident  in  different  databases,  is  brought  into  the 
assessments  framework.  Speaking  for  the  mathematical  coherency  of  a  given  model,  if 
inputs  are  not  understood  how  is  the  traceability  of  the  arc  of  policy  to  outcome 
explained?  Speaking  very  practically,  if  cost  data  is  not  included  in  the  assessment  how 
do  implementers  and  program  executives  know  which  activities  are  high  yield  and  which 
are  low  yield? 

(3)  What  is  Missing  from  an  Understanding  of  How  GPOI  is  Actualized?  The 
author  believes  that  the  GPOI  is  very  complete  in  its  approach  in  laying  out  approved 
activities  and  providing  a  mechanism  for  organizational  consent.  Additionally,  the 
COCOMS  (the  implementer  set  being  examined)  catalogue  the  execution  of  these  events 
in  their  operations  database  known  as  Theatre  Security  Cooperation  Management 
Information  System  (TSCMIS).  What  is  not  clear  is  how  this  data  inside  TSCMIS  is 
migrated  to  a  consolidated  GPOI  assessment  (Perry  2013). 

The  GPOI  has  furnished  an  assessment  model,  known  as  the  full  training 
capability  (FTC)  that  measures  its  first  objective,  which  is  essentially  is  “to  assist  partner 
nations  in  reaching  FTC.”  The  FTC  maps  directly  back  to  the  first  objective.  However, 
the  FTC  only  addresses  one  of  the  six  stated  GPOI  objectives.  Assuming,  these  six  are 
the  right  objectives,  a  framework  that  measures  seventeen  percent  of  its  objectives  can 
hardly  be  defended  as  complete.  What  is  needed  are  models  that  incorporate  the  other 
five  objectives. 

(4)  What  is  Missing  from  an  Understanding  of  GPOFs  Impact?  The  principal 
difficulty  with  the  current  GPOI  assessments  framework  is  that  the  outcomes  or  end  state 
of  the  GPOI  are  not  made  explicit.  If  the  outcomes  are  not  explicit,  then  how  are  they  to 
be  measured?  For  example,  body  counts  such  as  number  of  troops  trained  are  not 
outcomes,  they  are  objectives.  Measuring  number  of  troops  trained  is  no  doubt  an 
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excellent  MOP,  but  it  is  not  a  measure  of  effectiveness  (MOE).  A  corresponding  impact 
for  the  body  count,  what  this  thesis  refers  to  as  an  outcome  as  assessed  by  an  MOE, 
would  be  something  along  the  lines  of:  How  did  troops  perform  on  mission?  Or  how  do 
those  contribute  to  the  peacekeeping  capacity  of  a  certain  capability  level?  How  did  those 
troops  contribute  to  regional  security? 

As  placeholders  to  build  beta  assessment  models,  the  author  offers  suggested 
outcomes  to  be  measured  based  on  his  understanding  of  the  GPOI  based  on  existing 
official  program  documents  (see  Chapter  V).  However,  this  determination  should  be 
made  by  the  highest  level  of  the  GPOI  program  management  structure  with  appropriate 
input  and  oversight  of  outside  bodies  as  necessary. 

4.  Functional  Architecture 

Eunctional  architecture  captures  and  decomposes  all  the  required  functions  or 
activities  that  are  necessary  to  attain  the  system  requirements.  Thus  requirements  are 
translated  to  activities  that  can  then  be  translated  into  a  physical  architecture.  While  this 
understanding  holds  true,  a  broader  definition  of  functional  architecture  has  been  applied 
herein  (Buede  2000). 

As  the  functional  decomposition  in  this  framework  was  very  straightforward, 
additional  elements  were  added  to  the  architecture.  The  perhaps  traditional  view  of 
functional  architecture  was  modified  to  more  clearly  express  the  intent  of  the  proposed 
assessment  framework.  Eor  this  thesis.  Totem’s  view  of  functional  architecture  is  used, 
which  defines  it  as  a  framework  that  provides  capabilities  as  a  managed  business  process. 
The  emphasis  here  is  on  the  managed  process  that  encompasses  performers  (people  or 
organizations)  as  well  as  highlighting  links  to  other  system  elements  and  the  basis  of  each 
function  (Totem  Etd.  2011). 

a.  Proposed  Functional  Architecture 

Eigure  13  represents  the  proposed  functional  architecture  for  an  assessment’s 
framework  for  the  GPOI.  The  functional  architecture  as  proposed  is  essentially  a 
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functional  decomposition  placed  in  context  of  its  application,  its  applicators,  and  the 
elements  that  should  inform  and  govern  the  various  functions 

The  functional  decomposition  is  the  five  assessment  functions  that  describe  the 
various  types  of  assessment  that  must  occur  to  satisfy  the  requirements  as  developed  in 
the  capabilities  architecture.  These  five  functions  are  all  performed  by  a  series  of 
assessing  entities  that  have  been  notionally  labeled  program  management,  program 
accounting,  implementer  planning  cell,  in-country  assessment  teams,  and  outcome 
assessment  activities.  These  performers  are  more  explicitly  developed  in  the  supporting 
organizational  architecture  and  finally  specifically  assigned  to  a  real-world  person  or 
organization  in  the  actualizing  element  space. 


Figure  13.  GPOI  Functional  Architecture 


57 


The  five  assessing  functions  as  enumerated  should  be  informed  and  governed  by 
logical  standards.  For  example,  as  shown  in  Figure  13,  the  assess  output  function  should 
be  based  on  the  GPOI  objectives.  What  is  offered  is  a  one  for  one  mapping  that  will 
inform  the  assessment  models  themselves. 

Additionally,  the  system  functions  are  oriented  to  the  system  itself,  as  each  GPOI 
function  maps  to  the  portion  of  the  system  that  it  should  measure.  The  underlying  concept 
here  as  adapted  from  Santos  (see  discussion  in  Chapter  II  on  the  subject)  is  that  by 
viewing  the  system  in  question  as  a  process  traceability  is  maintained,  defined 
measurement  points  are  offered,  and  the  necessary  data  diversity  is  achieved.  The  key 
distinction  from  Santos’s  model  is  breaking  out  plans  and  policy  from  the  process  block 
as  a  separate  entity  for  assessment  (see  Figure  14).  As  previously  mentioned,  this  was 
done  to  specifically  assess  plans  and  policy,  but  also,  as  explained  more  fully  in  Chapter 
V,  because  it  is  not  necessary  to  conduct  a  plans  and  policy  assessment  every  time  and 
thus  it  can  be  omitted  when  measuring  an  individual  GPOI  event.  Additionally,  plans  and 
policy,  when  thought  of  in  terms  of  precedence,  should  initiate  the  process. 


Figure  14.  Viewing  the  GPOI  System  as  a  Continuous  Process 

(after  Santos  2011) 


b.  Achieving  the  Functional  Architecture 

The  functional  architecture,  as  expressed  in  Figure  13,  can  be  summarized  as 
follows:  enumerate  the  high  level  functions,  place  them  in  context  of  the  specific  system 
application  points,  articulate  the  governing  basis  of  the  functions,  and  assign  an 
organizational  element  to  achieve  them.  Hence,  the  functional  architecture  is  effectively  a 
blend  of  functions,  standards,  context,  and  implementers. 

To  achieve  the  functional  architecture,  one  more  layer  of  architecting  is  required. 
From  the  functional  architecture  view  supporting  data  application  and  organizational 
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architecture  should  be  developed.  The  purpose  of  pulling  these  two  layers  out  from  the 
functional  architecture  is  to  have  a  basis  for  developing  the  end  state  items  that  will  be 
required.  All  of  the  end  state  items,  also  referred  to  as  actualizing  elements,  can  generally 
be  binned  as  technical  products  or  organizational  products. 

5.  Data-Based  Architecture 

The  intent  of  the  data-based  architecture  is  to  enumerate  the  system  requirements 
that  are  necessary  to  achieve  the  information-based  elements  of  the  functional  hierarchy 
(Figure  13)  that  in  turn  achieves  the  capabilities  architecture  (Figure  12).  The  data-based 
architecture  describes  the  framework  for  properly  measuring  and  assessing  the  system. 
As  the  name  would  suggest,  this  architecture  is  focused  on  articulating  the  required  flow 
of  data  so  that  the  proper  assessment  models  can  be  built.  The  assessment  models 
themselves  that  make  up  the  data  architecture  are  discussed  in  Chapter  V. 

a.  Proposed  Data-Based  Architecture 

Figure  15  is  a  representation  of  the  data-based  architecture.  Four  layers,  or  levels, 
of  data  management  are  proposed.  The  first  is  the  GPOI  itself  system  from  which  the 
assessments  are  made  from  defined  measurement  points.  Comprising  the  second  level  are 
the  various  assessment  models  that  draw  upon  the  system  at  the  aforementioned 
measurement  points.  Data  synthesis  is  the  third  level  by  aggregating  and  lending 
coherency  to  the  various  assessment  models.  And  the  final  level,  or  the  end  state  of  the 
data-based  architecture,  is  the  output  of  system  understanding  for  consumption  by 
analytic  bodies  and  decision  makers. 
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A  Data  Based  Architecture  for  Assessments  in  Global  Peace  Operations  Initiative 
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Figure  15.  GPOI  Data-Based  Architecture 


h.  Implications  of  the  Data-Based  Architecture 

What  follows  is  discussion  on  the  important  takeaways  from  the  data-based 
architecture.  Specifically,  this  includes  the  elements  of  the  architecture  that  define  and 
inform  the  assessment  models  that  are  required  to  support  the  framework. 

(1)  The  Necessity  and  Importance  of  Assessment  Aggregation.  It  would  be 
convenient  if  the  Global  Peace  Operations  Initiative  could  be  assessed  using  a  unitary 
model.  However,  the  disparate  nature  of  the  data  collection  activities  and  storage 
mechanism  used  probably  preclude  this.  If  the  supporting  organizational  architecture 
details  a  division  of  labor  across  agency  boundaries  (e.g.,  DOD,  DOS,  DSCA,  and  UN 
DPKO)  then  it  is  likely  that  different  entities  are  making  assessments  at  different  parts  of 
the  system  and  likely  using  varied  networks  and  database.  The  implication  is  that  there  is 
likely  the  need  to  draw  upon  dissimilar  assessment  resources  and  synthesize  data. 
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(2)  The  Need  for  Data  Diversity.  As  discussed  in  Chapter  II,  depending  on  the 
question  being  answered  different  parts  of  the  system  will  be  need  to  be  measured. 
Hence,  the  importance  of  constructing  data  architecture  that  is  capable  of  measuring 
across  the  system.  Data  diversity,  which  is  defined  in  the  proposed  assessment  models  as 
making  measurements  in  each  part  of  the  system  (as  defined  by  process  flow  in  Figure 
14),  also  helps  maintain  traceability. 

(3)  The  Need  for  Defined  Measurement  Points.  As  discussed  throughout, 
mining  the  right  data  from  the  system  is  the  key  to  making  the  assessment  framework 
function.  It  need  not  be  overly  complicated  to  determine  the  correct  measurement  points. 
Measurement  points  can  be  determined  using  two  concepts.  The  first  is  decomposing  the 
requirements  to  determine  what  type  of  data  is  required.  And  the  second  concept  is 
traceability,  which  is  simply  expressed  by  matching  data  type  to  data  requirement.  For 
example,  when  building  the  output  assessment  model  (which  was  determined  to  be 
required),  its  various  metric  should  be  based  on  the  stated  program  objectives. 

6.  Organizational  Architecture 

The  organizational  architecture  is  the  non-technical  corollary  to  the  data-based 
architecture.  While  the  data-based  architecture  describes  information  flowing  through 
levels  of  the  system  using  various  assessment  mechanisms  the  organizational  architecture 
describes  information  flowing  through  the  system  using  various  organizational  nodes.  An 
organizational  node  is  defined  as  a  person,  agency,  or  entity  that  carries  out  an  activity  in 
support  of  a  requirement.  Again,  the  importance  of  expressing  both  data  and 
organizational  views  is  that  for  every  level  of  the  assessment  framework  there  is  required 
to  be  some  technical  mechanism  as  well  as  a  node  to  execute  it. 

a.  Proposed  Organizational  Architecture 

The  proposed  organizational  architecture,  as  expressed  in  Figure  16,  was  built 
using  the  OV-2  operational  node  connectivity  description  as  a  template.  The  OV-2  is 
adapted  from  a  Department  of  Defense  Architecture  Framework  (DODAF)  view  that 
expresses  the  system  in  terms  of  how  information  moves  from  various  nodes.  The  two 
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main  features  of  the  OV-2  are  the  nodes  themselves,  which  should  enumerate  who  is 
doing  what  activities  and  need  lines  that  are  defined  by  the  type  of  information  that 
moves  across  the  connections  (Dam  2006). 


An  Oreanizational  Architecture  for  Assessments  in  the  Global  Peace  Operations  Initiative 


Figure  16.  GPOI  Organizational  Architecture 


b.  Implications  of  Organizational  Architecture 

What  follows  is  discussion  on  the  important  takeaways  from  the  organization- 
based  architecture.  Specifically  discussed  are  the  characteristics  of  the  architecture  that 
define  and  inform  the  actualizing  elements  that  will  be  required  to  support  the 
assessments  framework.  In  this  case,  the  actualizing  elements  that  would  be  informed 
would  include,  but  not  be  limited  to,  processes  and  procedures  and  organizational 
resource  management. 

(1)  Organizational  Resource  Management.  The  point  of  enumerating  the 
various  nodes  is  to  make  sure  that  all  of  the  necessary  assessment  activities  are  explicitly 
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assigned  to  the  right  agency.  While  it  is  possible  that  multiple  nodes  could  be  serviced  by 
one  organization,  it  is  doubtful  given  their  disparate  nature  that  they  could  all  be  serviced 
effectively  by  any  one  entity.  Hence,  a  beta  organizational  resource  management  plan  is 
offered  in  Chapter  V  to  assist  the  parties  in  developing  the  appropriate  coordination. 

(2)  Information  Promulgation  Concerns.  The  architecture  recognizes  that  the 
nature  of  the  information,  especially  when  taken  in  totality  as  it  aggregates,  may  in  fact 
be  very  sensitive.  Thus,  there  likely  needs  to  be  a  node  in  the  system  that  controls 
distribution  of  assessments.  This  is  to  reiterate  that  assessments  are  not  carried  out  for 
their  own  sake  but  are  part  of  a  larger  decision-making  process. 

(3)  Process  Development.  A  benefit  of  expressing  an  organizational 
framework  is  that  it  lends  itself  to  building  necessary  process  models.  What  was  proposed 
in  Figure  16  is  only  a  high  level  abstraction.  But  if  it  were  to  be  expanded  with  its 
supporting  actualizing  elements  and  verified,  it  would  go  a  long  ways  towards  making 
explicit  the  processes  necessary  to  run  an  effective  assessments  program. 

B.  ARCHITECTURE  ADVANCEMENT 

The  architectures  as  presented  in  this  chapter  were  based  on  the  need  to  bridge 
void  that  existed  between  a  fuzzy  problem  space  and  a  defined  solution  space. 
Nevertheless,  there  is  likely  much  room  for  improving  the  completeness  and  accuracy  of 
the  proposed  architecture. 

1.  Implementing  the  Architecture 

The  mechanisms,  or  what  has  been  described  as  the  actualizing  elements,  must  be 
enumerated  and  developed.  The  enumeration  of  these  actualizing  elements  as  well  as 
their  partial  development  is  offered  in  Chapter  V. 

2.  Refining  the  Architecture 

Building  the  actualizing  elements  as  laid  out  in  Chapter  V  does  improve  the 
architecture.  However,  the  architecture  should  be  refined  in  two  ways.  The  first  would  be 
a  continued  spiral  development  (see  Figure  3)  between  researchers  and  end  users.  And 
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the  second  is  the  verification  and  validation  process  of  testing  and  employing  the 
framework  and  its  associated  products  when  the  sponsor  is  adequately  satisfied  with  the 
product. 
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V.  DEVELOPING  ACTUALIZING  ELEMENTS 


The  intent  of  this  chapter  is  to  further  the  proposed  assessment  framework  for  the 
GPOI  by  developing  the  next  level  of  abstraction  of  the  design  in  the  actualizing  element 
space  (see  Figure  10).  As  previously  defined,  the  actualizing  elements  are  those 
components  of  the  design  that  constitute  the  data-based  and  organizational  architectures. 
The  actualizing  elements  that  are  developed  in  this  chapter  are  principally  the  assessment 
models  themselves  and  an  organizational  resource  management  plan.  There  are  other 
necessary  actualizing  elements,  such  as  codified  processes  and  procedures,  which  are 
discussed  but  not  developed  to  the  same  level  of  detail. 

A.  SATISFYING  THE  DATA-BASED  ARCHITECTURE 

The  data-based  architecture  as  developed  in  Chapter  IV  (see  Figure  15)  yields 
requirements  for  the  eventual  assessment  models.  The  three  most  important  data-based 
requirements  can  be  summarized  as  follows:  (1)  it  necessary  to  have  defined  points  of 
system  measurement  (2)  at  least  five  different  assessments  will  be  necessary  (plans  and 
policy,  inputs,  processes,  outputs,  and  outcomes)  and  (3)  there  is  a  requirement  to 
aggregate  and  synthesize  these  five  assessments. 

I.  Making  Outcomes  and  Their  Linkages  to  Objectives  Explicit 

As  discussed  in  the  development  of  the  capabilities  architecture  in  Chapter  IV 
(see  Figures  II  and  12),  there  is  a  fundamental  need  both  to  state  clearly  the  outcomes  as 
well  as  to  measure  them.  The  mission  of  GPOI  is  clearly  stated.  However,  it  the  mission 
(or  end  state)  should  be  expressed  as  a  series  of  defined  outcomes.  Additionally,  once  the 
outcomes  are  enumerated,  supporting  objectives  must  be  clearly  linked  to  outcomes.  If  an 
objective  does  not  link  to  an  outcome,  then  it  probably  should  not  be  an  objective  at  all 
(Bureau  of  Political-Military  Affairs  and  Office  of  Plans,  Policy  and  Analysis  2013). 

a.  Outcomes  in  Context  of  the  System 

A  sound  assessment  framework  would  be  able  to  answer  an  array  of  questions, 
but  none  is  more  important  than:  “Did  the  GPOI  investments  matter?”  As  illustrated  in 
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Figure  17,  plans  and  policy  dictate  program  events  (or  lines  of  effort)  that  should  achieve 
objectives  in  order  to  realize  the  end  state  outcomes. 


REALIZE 


ACHIEVE 


OBJECTIVES 


DIRECT 


PROGRAM  EVENTS 


POLICY,  PLANS,  PROCESSES  & 
PROCEDURES 


Figure  17.  Defined  Outcomes  in  Context  of  the  GPOI  Program 

This  simple  and  straightforward  line  of  thinking  bears  an  important  implication 
for  the  design  of  an  assessment  framework:  both  the  levels  themselves  and  the  linkages 
between  the  levels  must  be  defined  if  the  operational  planning  and  follow  on  assessment 
are  to  be  coherent. 

b.  Proposed  Outcome  Definition  for  the  GPOI 

In  the  GPOI  Implementation  Guide  (GIG)  the  U.S.  Department  of  State  lays  out 
six  objectives  (see  Table  1)  for  the  program  to  achieve  during  phase  II.  All  six  of  these 
objectives  are  decidedly  measures  of  performances,  and  they  should  be  explicitly  defined 
as  such.  While  the  outcomes  or  MOEs  of  the  program  are  not  made  explicit,  the 
overarching  mission  is  clearly  articulated.  The  GIG  states  that,  “GPOFs  phase  II  mission 
is  to  enhance  international  capacity  to  effectively  conduct  UN  and  Regional  peace 
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operations...”  Hence,  the  two  formal  outcomes  for  assessment  (MOEs)  are  defined  as: 
(1)  Enhanced  Regional  Security  and  (2)  Successful  Execution  of  United  Nations 
Peacekeeping  Mission  (Bureau  of  Political-Military  Affairs  and  Office  of  Plans,  Policy 
and  Analysis  2013). 

These  two  MOEs  capture  the  GPOI  mission.  The  six  objectives  (MOPs)  all  map 
to  at  least  one  MOE  (see  Eigure  18). 


Measures  of  Performance 

(from  stated  GPOI  Objectives) 


Measures  of  Effectiveness 

(derived  from  stated  GPOI  Mission) 


i 


i 


Enhance  the  Peace  Operations  Capacity  of  Regional/ 


Eigure  18.  Einking  Objectives  (MOPs)  to  Outcomes  (MOEs)  (after  Bureau  of 
Political-Military  Affairs  and  Office  of  Plans,  Policy  and  Analysis 

2013) 


c.  Time  and  Space  Distinctions  between  Objective  and  Outcome 
Assessments 

With  the  objectives  and  outcomes  clearly  defined  as  well  as  their  linkages,  the 
practical  matter  of  their  assessment  must  be  considered.  In  Eigure  19,  it  is  suggested  that 
achievement  of  objectives  and  realization  of  outcomes  occur  at  fundamentally  different 
points  in  the  process  and  thus  should  be  assessed  separately. 
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Objectives  are  achieved  by  the  GPOI  program  through  a  series  of  approved 
events.  The  MOPs  should  be  assessed  during  or  directly  following  a  GPOI  event.  Indeed, 
every  GPOI  event  should  have  built  into  the  plan  a  means  of  assessing  the  degree  to 
which  the  objectives  were  achieved.  The  implementers  themselves  or  other  elements  of 
the  in-country  political-military  group  would  be  good  candidates  for  carrying  out  these 
assessments. 

OBJECTIVES  AND  OUTCOMES  LIKELY  REQUIRE 
SEPARATE  MEASUREMENT  POINTS 


GPOI  Stated 
Objectives 
are 

achieved  at 
the  Event 
Level 


Outcomes 

are 

achieved 
downstrea 
m  of  events 
either 
inside 
partner 
nation  or 
on  an  UN 
peace 
keeping 
mission 


Figure  19.  Fundamental  Space  and  Time  Difference  between  Objectives  and 

Outcomes 


The  outcomes  are  downstream  in  the  process,  and  their  measurement  opportunity 
is  likely  at  a  later  time  as  well  as  in  a  different  location  than  the  objective  measurement. 
The  implication  is  that  the  while  objectives  do  achieve  the  outcomes,  they  likely  require 
separate  measurement.  For  example,  the  “Successful  Completion  of  United  Nations 
Peacekeeping  Mission”  MOE  must  be  measured  either  on-mission  or  post  mission  (which 
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may  be  some  time  later  and  eertainly  in  a  different  plaee  than  where  the  troops  were 
trained).  Essentially,  the  outcomes  must  be  measured  when  and  where  they  occur;  hence, 
the  proposed  needs  for  separate,  but  compatible,  assessment  models  that  can  later  be 
aggregated. 

2.  Assessment  Aggregation 

As  discussed  in  the  development  of  the  data-based  architecture,  the  nature  of  the 
GPOI  system  necessitates  a  node  that  is  capable  of  aggregating  assessments.  In  order  to 
synthesize  these  assessments,  it  is  important  that  the  supporting  models  be  built  with 
compatibility  in  mind.  What  follows  is  a  discussion  on  a  proposed  assessment 
aggregation  model  and  how  it  could  be  implemented. 

a.  Partitioning  the  Assessment  Space 

The  five  required  assessment  models  can  be  binned  based  on  their  expected  time 
periods  as  well  as  at  their  organizational  level.  Figure  20  illustrates  that  the  assessments 
be  grouped  as  following:  (1)  plans  and  policy  assessment,  (2)  assessments  of  inputs 
(MOIs),  processes  (MOPr),  and  outputs  (MOPs)  and  (3)  outcome  assessments  (MOE). 

Plans  and  policy  assessment  naturally  occur  at  the  program  office  level  at  a 
defined  periodicity.  While  plans  and  policy  should  inform  the  rest  of  the  process  it  is 
unnecessary  to  reassess  every  time  a  GPOI  event  is  conducted.  Plans  and  policy 
assessment  might  be  undertaken  annually,  with  stakeholders  that  span  the  spectrum  from 
oversight  to  implementers. 
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GROUPING  ASSESSMENT  MODELS 


Assess  during/post 
regional/international 
peacekeeping  event 


.  „  -  ir  Assess  During/Post  GPOl  Event 

Assess  Periodically 


Occurs  at  Program 
Office  Level 


Occurs  at  the  Implementation  Level 


Occurs  at 
Strategic  Level 


Figure  20.  Proposed  Assessment  Model  Partitioning 


The  second  “bin”  at  the  implementer  level  is  understood  by  measuring  the  arc  of 
resource  input  (organizational  capacity  or  financials)  through  event  execution  to  objective 
achievement.  The  implementer  bin  is  explained  in  greater  detail  in  the  following  section 
(see  Figure  22). 

Finally,  and  most  problematic  of  all,  is  the  outcome  assessment.  For  reasons 
already  mentioned,  this  is  a  fundamentally  a  different  assessment.  While  other  work¬ 
around  metrics  may  be  introduced  as  necessary  to  infer;  the  desired  measurement  points 
will  be  that  of  GPOI  influenced  personnel  (e.g.,  directly  trained,  equipped  or  indirectly 
trained  via  the  train  the  trainer  events)  as  they  perform  on  assignment  in  either  regional 
security  assignments  or  on  United  Nations  peacekeeping  missions. 

The  distinct  nature  of  these  three  groups  of  assessments  precipitates  the 
following:  firstly  that  the  assessment  models  be  built  as  partitioned  above  so  that  they 
will  be  compatible  for  later  synthesis  and  secondly  that  a  central  node  be  responsible  for 
the  data  aggregation. 
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b.  Aggregation  at  a  Defined  Node 

The  previous  discussion  laid  out  a  notional  partitioning,  or  grouping,  of  the 
assessment  space.  Using  the  proposed  grouping  as  starting  point,  an  assessment 
aggregation  model  can  be  developed.  From  this  overarching  model  the  supporting  models 
can  be  derived. 

(1)  Aggregation  Node  Assessment  Model.  A  central  node  should  use  a  model 
like  the  one  suggested  in  Figure  21  to  bring  together  the  aforementioned  three  groups  of 
assessments.  The  central  node,  which  is  discussed  in  more  detail  later,  would  be  the 
person,  organization,  or  agency  that  is  assigned  the  responsibility  of  synthesis  across  the 
measurement  space. 

The  idea  is  that  a  diverse  set  of  implementers  (such  as  COCOMS,  contractors,  and 
State  Department  regional  bureaus)  conduct  and  assess  a  large  number  of  events  in 
different  partner  countries.  These  events  are  assessed  from  the  arc  of  input  to  objective 
using  a  common  format  and  forwarded  to  the  aggregation  node.  At  the  aggregation  node 
these  events  are  put  against  one  of  the  stated  end  states  (outcomes  as  assessed  by  MOEs). 
As  previously  discussed,  the  all-important  outcome  assessment  is  likely  conducted 
separately.  The  important  feature  of  the  model  is  that  the  outcomes  as  measured  are 
linked  back  to  the  objectives  that  were  supposed  to  realize  them,  which  are  in  turn  linked 
back  to  processes  (activities)  that  achieved  the  objectives,  which  are  in  turn  linked  back 
to  the  inputs  that  enables  the  processes,  which  is  finally  linked  back  to  overarching  plans 
and  policy  that  directs  the  whole  program.  (See  Figure  21). 
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Modeling  Notional  Assessment  Aggregation 

1^'  Event  Assessment 


Occurs  at  Program 
Office  Level  as 
necessary 


Y 

Events  Assessments  from  Regional  Implementers 


J 


MOE  Assessment 
Occurs  separately 
from  Event 
Assessments 


Figure  21 .  Model  of  a  Proposed  Assessment  Aggregation 


(2)  Supporting  Implementor  Node  Assessment  Model.  Using  the  proposed 
assessment  aggregation  model  (see  Figure  21)  as  a  basis  a  notional  assessment  model  at 
the  implementor  level  is  suggested  (see  Figure  22).  The  idea  is  as  the  implementor  set  is 
planning  a  GPOI  event,  the  assessment  model  is  being  built  in  parallel. 

By  stitching  together  an  input  assessment  model,  a  process  assessment  model,  and 
an  output  assessment  model.  While  the  outcomes  are  measured  separately,  the 
implementer  should  indicate  which  of  the  stated  outcomes  is  being  addressed. 
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Modeling  Notional  Event  Assessment  at 
Implementer  Level 


Implementers  puts  the  appropriate  MOIs, 
MOPrs,  and  MOPs  models  together  to  assess  a 
GPOl  Event(s) 


Implementers  states  which  MOE  are 
addressed  but  MOES  ARE  NOT  MEASURED  BY 
IMPLEMENTER  AS  PART  OF  THE  EVENT 
ASSESSMENT.  Measurement  of  MOEs  occurs 
downstream,  likely  by  higher  (or  outside 
authority) 


Figure  22.  Model  of  a  Notional  Event  Assessment  at  the  Implementation  Level 


3.  Developing  Defined  Metrics  for  the  Assessment  Models 


Realizing  assessment  models  (as  proposed  in  Figures  21  and  22)  requires  one  last 
layer  of  abstraction,  which  is  assigning  specific  metrics  for  each  of  the  defined 
assessment  models.  What  follows  is  the  proposed  basis  of  specific  metrics  based  on  the 
architecture  development. 


a.  A  Note  on  Metrics:  Context  and  Development  Process 

The  metrics  are  considered  to  be  a  component  of  a  complete  assessment 
framework  and  not  the  assessment  framework  itself.  Additionally,  as  illustrated  in  Figure 
23,  metrics  should  be  defined  towards  the  end  of  the  design  process  to  fulfill  the 
assessment  models. 
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ENGINEERING  THE  RIGHT  METRICS 
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Figure  23.  Developing  the  Right  Metrics  for  the  GPOI 


The  benefit  to  this  design  process  is  that  it  helps  the  engineer  cut  through  the 
clutter  of  an  almost  endless  list  metrics  to  choose  from.  By  having  defined  requirements 
as  to  what  the  assessment  models  should  measure,  metrics  can  be  chosen  in  a  way  that 
maps  back  to  the  original  problem  understanding. 

b.  Metric  Selection  Basis 

The  architecture  development  informs  the  metric  selection  process  in  two  ways. 
First,  it  clearly  defines  the  specific  points  in  the  system  that  need  to  be  measured  and 
secondly  it  addresses  what  questions  should  be  asked.  Using  this  framework,  as 
expressed  in  Appendix  K,  metrics  can  be  assigned. 

The  takeaway  from  the  proposed  metrics  requirements  is  that  most  of  the  metrics 
can  be  derived  from  the  GPOI  program  itself.  For  example,  the  objectives  are  made 
explicit;  the  MOPs  can  be  built  to  answer  those  objectives. 
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The  metrics  that  will  prove  the  most  challenging  to  develop  will  be  those  for  the 
outcome  assessment.  The  difficulty  will  be  choosing  a  measure  of  effectiveness  that  can 
linked  back  to  stated  measures  of  performance. 

B.  SATISFYING  THE  ORGANIZATIONAL  ARCHITECTURE 

The  organizational  architecture  as  developed  in  Chapter  IV  (see  Figure  16) 
defines  the  various  nodes  in  the  system  by  their  responsibilities  and  relationships  to  one 
another.  To  review,  nodes  are  defined  as  the  responsible  organization  or  person  that 
performs  the  necessary  operational  activities.  The  organizational  architecture  is 
developed  in  non-solution  specific  terms;  that  is  it  outlines  that  someone  or  some  group 
needs  to  perform  certain  actions  if  the  corresponding  data-based  architecture  is  to  be 
realized. 

The  next  step  in  development  is  to  develop  a  suggested  organizational  resource 
management  plan  that  can  satisfy  the  aforementioned  organizational  architecture.  The 
organizational  resource  management  plan,  which  should  possess  coherency  with  the  rest 
of  the  architecture,  should  be  realized  by  a  set  of  memoranda  of  understanding  (MOU)  or 
memoranda  of  agreement  (MO A)  between  the  various  agencies.  If  this  last  step  of 
codification  is  completed,  then  it  is  more  likely  that  the  necessary  buy-in  and 
understanding  has  been  achieved  throughout  the  enterprise.  Figure  24  illustrates  this 
general  approach,  which  is  to  drive  towards  the  end  state  of  defined  roles  and 
responsibilities  as  part  of  the  larger  systems  engineering  effort. 
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Figure  24.  Developing  the  Right  Organizational  Resource  Application  for  GPOI 

Assessments 


1.  Notional  Organizational  Resource  Management  Model 

Appendix  M  proposes  specific  assignment  of  assessment  responsibilities.  The 
notional  assignments  are  based  on  the  node  that  is  best  equipped  to  carry  out  the 
necessary  operational  activities.  The  intent  is  to  limit  unnecessary  duplication  of  data 
gathering  and  analysis  or  what  is  commonly  referred  to  in  the  intelligence  community  as 
“stove  piping.”  It  is  hoped  that  by  leveraging  the  different  views  of  different  nodes  within 
the  system  that  a  better  assessment  can  be  achieved.  This  is  of  course  limited  by  the 
ability,  and  will,  of  the  various  organizations  to  partner  effectively. 

The  division  of  labor  as  proposed  in  Appendix  L  suggests  splitting  the  work 
across  three  principal  groups.  The  first  group  is  the  GPOI  program  manager  who  would 
be  responsible  for  coordinating  the  various  assessment  activities  as  well  as  necessary  data 

synthesis.  The  second  group  is  the  implementer  set  that  focuses  on  arc  of  money  to 
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objective  attainment.  And  the  third  group  is  the  outside  agencies  that  provide  validation 
assessment.  This  third  group  is  suggested  to  be  comprised  of  the  UN  DPKO  to  assess  the 
performance  of  GPOI  troops  on  peacekeeping  missions  as  well  as  an  appropriate  agency 
to  assess  regional  security. 

This  division  of  labor  can  also  be  viewed  through  the  lens  of  the  SE- V  model  (see 
Appendix  C).  What  is  proposed  is  that  the  program  management  focuses  on  the  tops  of 
the  V,  which  is  the  validation  level.  While  the  implementer  set  would  focus  on  the 
bottom  of  the  SE-V,  which  is  the  verification  level. 

2.  A  Note  on  the  Use  of  Outside  Agencies 

As  discussed  throughout  this  thesis,  it  is  firmly  believed  that  assessing  the  GPOI 
is  very  problematic  as  long  as  the  end  states  are  not  made  explicit  and  measured.  The  first 
part  is  less  complicated,  as  logical  end  states  can  be  translated  from  the  stated  mission 
and  linked  back  to  the  objectives  (see  Eigure  18).  However,  as  previously  discussed  it  is 
difficult  for  an  assessments  team  to  measure  outcomes  as  they  occur  outside  the  time  and 
space  purview  of  the  GPOI  (see  Eigure  19),  which  is  precisely  why  it  is  proposed  that 
outside  agencies  be  polled  to  measure  the  stated  outcomes. 

It  is  suggested  that  the  UN  DPKO  or  other  member  of  the  United  Nations  body  be 
polled  for  the  performance  of  GPOI  trained  troops  on  mission.  The  case  is  here  is  simply 
is  that  the  end  user,  and  not  the  designer,  who  should  be  the  principal  evaluator  of  the 
system.  Eor  less  defined  notional  outcome  of  “enhanced  regional  security,”  it  is  likely 
that  other  members  of  the  US  Government  will  track  regional  stability.  These  other  US 
Government  entities  could  be  solicited  as  the  basis,  at  least  in  part,  for  an  assessment  of 
any  improvement  or  decrement  in  the  region. 

The  third  party  assessors  of  outcomes  offer  three  practical  advantages:  (1)  the 
third  party  has  the  potential  to  addresses  a  situation  where  the  GPOI  has  insufficient 
access  and/or  capacity  to  measure;  (2)  working  with  the  end-users  builds  in  a  feedback 
loop  into  the  system  for  continual  improvement;  (3)  as  third  parties  they  perhaps  have  a 
built  in  degree  of  objectivity  as  to  the  real  “so  what”  of  the  GPOI.  This  likely  strengthens 
the  case  to  be  made  for  GPOI’s  impact. 


77 


C.  ADDITIONAL  ENABLING  ACTUALIZING  ELEMENTS 

What  will  really  drive  the  assessments  framework  is  a  sound  and  a  complete  set  of 
models  with  corresponding  organizational  resource  management  plan  to  enact  them. 
However,  there  are  likely  additional  elements  that  should  be  developed  in  order  to  fully 
round  out  and  effectively  adopt  any  architecture. 

Although  not  expressly  developed  in  this  thesis,  what  follows  is  a  brief  discussion 
of  these  additional  enabling  actualizing  elements  that  would  likely  require  development. 
They  are  not  developed  because  they  are  outside  the  scope  of  research,  and  secondly  they 
are  downstream  of  an  agreed  upon  framework.  The  principal  “other  actualizing  elements” 
to  be  developed  and  defined  would  be  corresponding  processes  and  procedures  and 
overarching  doctrine. 

On  the  processes  and  procedures  issue  the  idea  would  be  to  incorporate  any 
adopted  changes  of  the  assessment  methodology  into  the  GIG.  The  most  important  one 
would  perhaps  be  the  need  to  include  assessment  model  as  part  of  event  planning.  Also, 
the  flow  of  assessments  throughout  the  system  would  need  to  be  made  explicit. 

The  adoption  or  reference  towards  a  standing  doctrine  would  be  beneficial.  This 
proposed  framework  is  built  from  SE  principles.  Grounding  in  field  tested  assessment 
principles  would  be  beneficial  to  both  in  the  implementation  and  improvement  on  any 
assessment  framework.  For  example,  the  Army  Field  Manual  5.0  might  be  a  useful 
reference  to  orient  the  entire  team  as  to  role  of  assessments  within  the  broader  context  of 
operations  being  conducted  (The  Department  of  the  Army  2012). 
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VI.  CONCLUSIONS,  RECOMMENDATIONS,  AND  FUTURE 

WORK 


The  intent  of  this  chapter  to  take  stock  of  the  research  conducted  and  explored 
four  lines  of  questioning.  The  first  is  to  examine  to  what  degree  the  research  fulfilled  its 
stated  intent.  Secondly,  the  intent  is  to  draw  final  conclusions,  both  as  they  pertain  to  the 
GPOI  itself  as  well  as  those  of  a  more  generalizable  nature.  Thirdly,  based  on  the 
aforementioned  conclusions  and  totality  of  the  research  the  intent  is  to  make 
recommendations  to  the  sponsor  for  suggested  means  of  improving  the  assessments  for 
the  GPOI.  Fourth,  and  finally,  a  discussion  is  offered  on  how  the  work  contained  herein 
might  be  advanced  by  the  future  research  of  others. 

A.  HOW  FAR  DID  THE  RESEARCH  GET? 

The  progress  of  this  research  is  assessed  against  the  stated  research  questions  and 
objectives.  For  this  thesis  a  primary  research  question  as  well  as  a  secondary  research 
question  was  posed.  Based  on  those  two  questions,  four  SE  product  specific  objectives 
were  developed.  What  follows  is  a  brief  restatement  of  the  original  research  questions 
and  objectives  (see  Chapter  I)  and  a  discussion  as  to  their  level  of  attainment. 

1.  Research  Question  Resolution 

The  research  questions  were  laid  out  as  a  way  to  give  an  overall  direction  for  the 
thesis.  They  speak  to  the  problem  at  a  high  level  of  abstraction  and  were  intentionally 
broad  as  it  was  unclear  how  the  problem  and  subsequent  solution  design  would  develop. 

a.  Original  Research  Questions 

The  primary  and  secondary  research  questions  are  enumerated  below.  The  first 
speaks  what  was  thought  to  have  been  the  chief  difficulty  in  assessing  the  GPOI,  which  is 
its  high  level  of  complexity.  The  second,  which  was  contingent  on  answering  the  first, 
speaks  to  the  assumed  need  to  develop  some  mechanism  to  assess  the  system. 

(1)  Primary  Research  Question.  The  primary  research  question  of  this  thesis 
was:  can  the  complexity  of  assessing  the  GPOOI  be  understood  and  managed? 
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(2)  Secondary  Research  Question.  The  secondary  research  question  of  this 
thesis  was:  Given  a  structured  problem  space,  how  should  the  supporting  assessment 
models  be  built? 

b.  Were  the  Research  Questions  Answered? 

It  is  generally  difficult  to  say  to  what  degree  the  research  questions  were 
answered  as  this  architecture  is  not  validated  (that  is  put  in  use  with  an  opportunity  to 
assess  its  worth).  It  would  perhaps  be  more  useful  to  address  to  what  degree  the  questions 
were  addressed.  By  answering  this  question,  the  groundwork  is  laid  for  a  cogent  future 
works  enumeration. 

(1)  Was  the  Primary  Research  Question  Answered?  It  is  difficult  to  determine 
to  what  degree  the  GPOI  complexity  is  understood  and  can  be  managed  because  there  is 
no  answer  key.  Achieving  problem  insight  was  addressed  by  trying  to  organize  and 
define  its  elements  and  managing  its  complexity  was  addressed  by  offering  a  series  of 
processes  that  lent  a  degree  of  structure  as  well  as  way  to  improve  future  iterations. 

Problem  understanding  was  attempted  via  standard  SE  practices.  The  different 
stakeholders  with  their  sometimes  competing  interests  were  catalogued.  Systems 
architecting  was  applied  to  give  the  problem  space  and  solution  space  a  degree  of 
structure. 

Articulation  of  the  problem,  no  matter  how  incomplete,  is  probably  an 
improvement  over  none  as  at  least  a  point  of  departure  has  been  developed  from  which  a 
more  meaningful  conversation  between  stakeholder  and  developer  can  be  held. 
Additionally,  while  a  more  robust  architecture  can  most  assuredly  be  developed,  the  one 
proposed  does  lay  out  what  are  believed  to  be  major  elements  and  their  interaction  with 
each  other. 

(2)  Was  the  Secondary  Research  Question  Answered?  This  question  was 
answered  partially.  The  proposed  beta  framework  provides  a  means  to  construct  the 
specific  metrics  that  would  be  traceable  to  the  stated  objectives  and  outcomes.  The 
general  approach  is  believed  to  be  sound  as  well  as  the  product  development  to  the 
degree  to  which  it  was  accomplished.  However,  the  research  stopped  short  of  proposing 
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all  of  the  specific  metrics  that  would  be  needed,  and  the  framework  itself  is  a  work  in 
progress  that  would  likely  benefit  from  some  more  iterative  design.  Hence,  the  focus  of 
future  work  would  be  a  refinement  of  the  architecture  and  development  of  what  this 
thesis  would  refer  to  as  the  fifth  and  final  layer  of  abstraction,  which  is  the  metrics 
themselves  (see  Figure  23). 

2.  Research  Objectives  Achievement 

What  follows  is  a  brief  explanation  of  the  original  research  objectives  and  a 
suggested  evaluation  as  to  the  level  of  their  attainment. 

a.  Original  Research  Objectives 

At  the  outset  of  the  research,  it  was  determined  that  the  problem  and  subsequent 
solution  design  would  be  developed  using  a  series  of  systems  engineering  processes  and 
systems  architecting  methodologies.  With  a  general  approach  in  mind,  the  two  research 
questions  were  decomposed  into  four  research  objectives  that  sought  to  translate  the 
problem  into  a  series  of  SE  artifacts  that  could  then  be  further  developed  into  an 
architecture  and  eventually  specific  mechanisms  to  actualize  a  solution.  In  Chapter  I,  four 
research  objectives  were  laid  out:  Expressing  the  GPOI  as  a  System,  Developing  a  GPOI 
Assessments  Eramework,  Developing  Supporting  Assessment  Models,  and 
Recommendations  for  the  GPOI  Assessments  Program 

b.  Research  Objective  Attainment 

What  follows  is  a  discussion  of  the  four  research  objectives  and  an  evaluation  as 
to  the  degree  of  their  attainment 

(1)  Expressing  the  GPOI  as  a  System.  As  mentioned  a  translation  effort  was 
undertaken  that  attempted  to  transform  understanding  of  the  GPOI  from  various  sources 
into  SE  artifacts.  However  this  was  primarily  done  from  the  specific  vantage  point  of 
assessing  the  GPOI.  To  more  fully  address  the  objective  of  expressing  the  GPOI  as  a 
system,  a  more  ambitious  modeling  effort  should  be  undertaken.  Another  thesis  was 
conducted  in  parallel  with  this  one  (see  Appendix  A)  that  viewed  the  GPOI  as  a  system  of 
systems. 
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(2)  Developing  a  GPOI  Assessments  Framework.  An  assessment  framework 
was  developed  for  the  GPOI  (see  Chapter  IV).  The  degree  to  whieh  it  is  valid  and  ean  be 
adopted  or  further  refined  and  then  adopted  is  still  to  be  determined.  The  arehiteeture  was 
developed  via  what  is  believed  to  be  a  eogent  and  elearly  artieulated  methodology. 

(3)  Developing  Supporting  Assessment  Models.  This  objeetive  was  partially 
aehieved.  The  supporting  models  and  proeess  for  satisfying  the  arehiteetures  of  the 
aforementioned  assessment  framework  were  developed.  However,  speeific  metrics  were 
not  assigned  to  them.  It  was  decided  to  place  that  effort  outside  of  the  scope  of  this 
research.  Assignment  of  metrics  would  be  best  addressed  after  the  sponsor  is  satisfied 
with  the  framework  first. 

(4)  Recommendations  for  the  GPOI  Assessments  Program.  Beyond  the 
proffered  framework  and  supporting  models,  this  thesis  did  yield  specific 
recommendations  for  the  GPOI  assessments  program.  These  recommendations  are  laid 
out  later  in  this  chapter. 

B.  CONCLUSIONS 

What  follows  is  a  discussion  as  to  what  are  believed  to  be  the  most  salient 
conclusions  of  this  research  effort  as  it  applies  to  addressing  the  need  for  an  approved 
assessment  of  the  GPOI. 

1.  Traceability  Is  Key 

Returning  to  the  problem  definition  phase  (see  Chapter  III)  the  stakeholders 
concerns  can  be  summarized  as  a  desire  to  be  able  to  justify  the  program  and  a  need  to 
provide  a  feedback  mechanism  for  course  corrections.  Both  of  these  concerns  require  that 
an  arc  be  drawn  between  investments  (money  or  other  organizational  resources)  and 
outcomes  or  “so  what?”  questions.  To  the  author’s  knowledge  there  is  no  other  way  to 
accomplish  either  the  justification  or  feedback  mechanism  without  traceability. 

Traceability  should  be  achieved  by  thinking  of  a  GPOI  event  as  a  process  (see 

Figure  14)  and  measuring  it  at  discrete  intervals.  It  is  necessary  that  all  of  the 

measurements  have  defined  linkages.  In  the  proposed  model,  the  inputs  (MOI)  drive  the 

processes  (MOPr),  which  in  turn  produces  outputs  (MOPs),  which  finally  realizes 
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outcomes  (MOEs).  An  effective  assessment  framework  can  be  articulated  in  ways  other 
than  proposed  in  this  thesis,  but  for  it  to  be  valid,  it  would  likely  have  to  provide  for  a 
degree  of  traceability. 

2.  Aperture  Matters 

Whether  one  is  trying  to  assess  a  specific  assessment  element  or  develop  a 
comprehensive  framework,  what  is  often  necessary  is  to  take  a  wider  view.  By  increasing 
the  scope  of  analysis,  other  elements  and  their  connections  are  illuminated.  In  the  case  of 
developing  the  framework,  taking  a  broader  view  caused  the  designer  to  consider 
elements  such  as  assessing  plans  and  policies  as  well  as  processes.  These  may  be  scoped 
out  of  the  final  framework  to  be  adopted,  but  an  intentional  exclusion  is  much  preferable 
to  leaving  them  out  due  to  lack  of  understanding  as  to  their  place  in  the  system. 

3.  Necessity  of  Solution  Organization 

Managing  complexity  and  thinking  holistically  is  difficult.  If  the  designer  is  to 
take  on  a  broader  perspective,  then  there  must  be  some  mechanism  to  handle  the 
increased  depth  (layers  of  architecture)  and  breadth  (number  of  elements).  Defined 
frameworks,  like  the  one  adapted  from  Totem  in  this  thesis,  are  useful  in  lending 
structure  to  an  otherwise  unwieldy  problem. 

4.  Process  and  Procedure  Matters 

While  this  thesis  in  not  a  study  in  usability,  the  development  of  products  was  done 
with  the  end  user  in  mind.  The  definition  of  end  user  that  has  been  used  in  different 
contexts  is:  when  taking  an  extra-system  view  the  ultimate  end  user  of  GPOI  is  the 
United  Nations  and  the  partner  nation/region  and  when  taking  an  inter-system  view  the 
end-users  of  the  assessment  framework  are  those  persons  that  occupy  the  implementer 
and  program  management  level  inside  the  GPOI. 

The  end  products  herein  that  were  proposed  are  assessment  models  and  a  notional 
resource  management  model.  However,  in  order  to  best  employ  the  assessment 
framework  the  end  users  would  benefit  from  clearly  articulated  processes  and  procedures. 

These  processes  and  procedures  can  be  developed  easily  enough  from  a  completed 
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architecture  that  should  account  for  both  technical  and  non-technical  elements.  The 
benefit  of  an  architecting  approach  is  that  keeps  the  user  in  mind  during  the  development 
and  hopefully  translates  to  a  design  that  can  be  better  actualized. 

C.  RECOMMENDATIONS  FOR  GPOI  ASSESSMENTS 

The  intent  of  this  section  is  to  communicate  a  series  of  recommendations  to  the 
sponsor  aimed  at  improving  the  state  of  assessments  in  the  GPOI.  The  suggestions  are 
binned  into  two  categories:  short  term  recommendations  and  long  term  recommendations. 
The  short  term  recommendations  are  those  that  are  believed  to  assist  the  sponsor  in  the 
foreseeable  future  if  adopted.  The  long-term  recommendations  are  aimed  at  the 
development  of  a  better  and  hopefully  more  enduring  solution. 

1.  Short  Term  Recommendations 

The  approach  as  outlined  in  this  thesis  aims  to  take  in  the  problem  in  its  totality 
and  develop  a  coherent  family  of  solutions.  While  it  is  believed  that  an  “engineered” 
assessments  framework  would  likely  be  the  most  complete  and  defensible,  a  great  benefit 
can  be  realized  by  going  after  “low  hanging  fruit.”  This  approach  is  more  of  a  heuristics 
or  best  practices  course  of  action,  as  opposed  to  the  more  rigorous  development  process 
as  discussed  throughout  this  thesis.  What  follows  is,  in  the  opinion  of  the  author,  the 
higher  yield  short  term  course  of  actions  that  the  sponsor  could  pursue  in  order  to 
improve  GPOI  assessments. 

a.  Achieve  Basic  Traceability:  Make  Outcomes  and  Objectives  Explicit  and 
Then  Measure  Them 

Adopting  an  overarching,  rationally  developed  framework  is  the  best  long  term 
solution.  However,  based  on  the  ill-defined  nature  of  the  current  assessments,  they  can  be 
dramatically  improved  just  by  adding  a  limited  degree  of  traceability. 

The  first  step  is  to  make  explicit  both  what  the  program  hopes  to  achieve 
(outcomes)  and  how  it  will  achieve  them  (objectives).  As  discussed  in  Chapter  V,  as  a 
point  of  departure,  the  two  suggested  notional  outcomes  could  be  adopted  (Figure  18). 
The  six  stated  objectives  (Table  1)  already  trace  neatly  to  those  notional  outcomes. 
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With  objectives  and  outcomes  made  explicit,  the  next  step  would  be  to  measure 
them.  The  first  objective,  full  training  capability,  is  already  being  measured,  so  the  other 
five  would  have  to  be  addressed.  Many  of  these  are  being  executed  and  catalogued,  but 
not  necessarily  aggregated  into  an  overall  GPOI  assessment.  For  example,  GPOI  events 
in  the  USSOUTHCOM  AOR  train  peacekeepers  (Phase  II  objective  No.  2)  and  provide 
support  for  deploying  units  (objective  No.  3).  Both  of  these  events  are  executed  in 
accordance  with  GPOI  polices  and  are  catalogued  in  TSCMIS.  The  point  is  perhaps  the 
need  for  another  five  assessments  can  be  sidestepped  by  performing  a  bit  of  an 
archeology  project  and  pulling  from  the  databases  of  past  events. 

This  still  leaves  the  issue  of  assessing  outcomes.  While  still  difficult,  it  is  made 
achieve  able  by  making  them  explicit.  The  State  Department  should  attempt  to  solicit 
performance  reviews  from  the  UN  DPKO  for  GPOI  trained  missions.  As  to  the  regional 
security  outcome,  it  is  suspected  that  between  the  various  agencies  working  in  the  AOR, 
perhaps  one  of  them  is  assessing  country  or  regional  security  that  could  be  used  as  an 
assessment  in  lieu  of  an  engineered  GPOI  outcome  measurement. 

This  might  indeed  be  a  rough-hewn  assessment,  but  it  would  allow  the  program  to 
make  statements  along  the  lines  of:  “the  GPOI  program  executed  these  events,  which 
achieved  these  stated  objectives,  which  contributed  to  the  realization  of  these  outcomes 
as  measured.”  As  far  as  the  author  is  aware,  that  fundamental  connection  is  generally 
lacking  in  the  GPOI  process.  This  would  be  a  significant  first  step  in  achieving  a  degree 
of  coherency  throughout  the  process  as  it  would  provide  justifiable  assessments  as  well  as 
help  to  re-orient  the  organization  to  the  centrality  of  traceability. 

b.  Elevate  Discussion  to  the  Program  Management  Level 

Even  modest  changes,  and  certainly  significant  ones,  are  unlikely  to  take  root 
without  the  buy  in  from  the  program  management  level.  The  framework  as  presented  in 
this  thesis  is  reliant  on  the  program  management  level  either  conducting  or  delegating 
key,  enabling  functions. 

What  was  suggested  in  the  previous  section,  basic  traceability,  as  well  as  the 
larger  suggestion  of  a  framework  can  be  understood  as  changes  in  organizational 
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processes.  While  the  bulk  of  the  work  will  likely  occur  at  the  implementer  level,  it  is 
assumed  that  policies  changes  require  the  buy  in  from  program  management. 

c.  Balance  the  Technical  with  the  Non-Technical 

Throughout  this  thesis  different  ways  of  thinking  about  the  problem  have  been 
offered,  but  if  pressed  to  offer  the  most  useful  re-orientation,  the  author  would  suggest 
this:  the  problem  is  fundamentally  a  balance  between  the  technical  and  the  non-technical. 

There  might  be  a  desire  to  have  a  unitary  measurement  model  that,  if  executed  by 
an  implementer,  would  satisfy  the  assessment  requirements.  That  of  course  is  principally 
a  technical  solution.  That  line  of  thinking  might  lead  to  a  bigger  and  perhaps  a  slightly 
improved  “stovepipe.” 

The  degree  to  which  the  non-technical  solution  can  be  developed  is  the  degree  to 
which  the  technical  problem  can  be  obviated.  That  is,  if  other  responsible  agencies  can  be 
tapped  for  analogous  assessment  data,  then  the  measurement  requirements  of  the 
implementer  are  correspondingly  reduced.  This  is  why  (see  longer  term  solution) 
complementary  data-based  and  organizational-based  architectures  are  suggested.  But  for 
the  short  term,  any  basic  inter-agency  collaboration,  or  just  use  of  existing  internal 
organizational  resources  may  yield  good  benefit. 

2.  Long-Term  Recommendations 

What  follows  is  a  discussion  on  the  course  of  action  that  the  author  believes  the 
sponsor  should  consider  if  they  are  pursue  a  more  complete  and  more  enduring  solution 
space  to  the  problem  of  assessing  the  GPOI.  Generally  the  recommendations  can  be 
binned  into  two  categories.  The  first  is  a  call  to  reconsider  how  the  problem  is  perceived. 
And  the  second  is  the  suggestion  of  building  a  more  complete  solution,  using  a  clear  and 
defined  methodology. 
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a.  Think  about  Your  Thinking 

As  discussed  in  Chapter  II,  the  GPOI  can  be  viewed  as  an  unperceivable  system; 
wherein  the  key  attributes  of  such  a  system  are  its  immense  scope  and  complexity  and 
opaqueness  of  many  of  its  underlying  interactions. 

Recognizing  the  GPOI  for  what  it  is  should  direct  the  designer  towards  a  few 
imperatives.  The  first  is  a  fundamental  need  to  think  holistically.  As  this  applies  to  GPOI, 
it  is  fundamentally  difficult  to  understand  any  one  element  without  widening  the  field  of 
view  so  as  to  capture  its  interactions  with  other  elements.  The  second  imperative  is  the 
need  to  manage  the  resulting  complexity. 

Thinking  at  this  level  of  abstraction  is  more  effective  with  a  systems  architecting 
point  of  view  than  a  pure  “engineering”  frame  of  mind.  Traditional  engineering  is  more 
or  less  an  optimization  exercise,  which  as  Richard  Balling  (1999)  points  out  requires  a 
well-defined  problem  space  with  articulated  constraints.  As  GPOI  is  in  fact  an  ill-defined 
problem,  systems  architecting  is  a  more  appropriate  approach,  as  it  gives  structure  to 
problem  space.  And  as  opposed  to  traditional  optimization,  systems  architecting  seeks 
more  to  hold  the  many  disparate  elements  in  balance. 

All  this  is  to  say  that  the  assessing  the  Global  Peace  Operations  Initiative  is  an 
immensely  complicated  undertaking.  GPOI  has  this  in  common  large  information 
technology  and  aerospace  projects  and  could  similarly  benefit  from  systems  architecting 
methods. 

b.  Cooperation  as  a  Major  Constraint 

As  mentioned  multiple  times,  the  solution  must  include  an  organizational 
component  to  complement  at  least  any  technical  solution.  Specifically,  the  stakeholder  set 
must  think  in  terms  of  inter  versus  intra  organizational  resource  management. 

Recalling  the  discussion  on  limitations,  (see  Chapter  III)  realize  that  any  modified 
assessments  framework  will  likely  face  significant  budgetary  constraints.  Thus,  it  is 
probably  not  realistic  to  propose  doing  more  with  the  same.  Instead,  doing  more  (more 
complete,  higher  veracity  assessments)  with  more  (other  partners)  is  more  realistic. 
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Even  if  resource  constraints  did  not  dictate  reaching  out  to  partners,  the 
coordination  should  be  executed  for  the  sake  of  assessment  quality.  The  stated  GPOI 
mission  (and  what  this  thesis  notionally  refers  to  as  an  outcome)  is  to  provide  trained 
troops  that  can  succeed  on  United  Nations  peacekeeping  missions;  should  not  that 
outcome  be  assessed  at  least  in  part  by  the  end-user,  which,  in  this  system,  appears  to  be 
the  United  Nations?  The  same  could  be  said  for  the  need  to  measure  regional  security. 

c.  Negotiate  a  Working  Framework 

Using  the  framework  developed  in  this  thesis  as  a  beta,  the  framework  should  be 
further  improved  and  tailored  to  the  sponsor’s  needs.  The  methodology  for  doing  so  as 
previously  discussed  would  be  to:  validate  the  problem  statement,  modify  the  capability 
architecture  as  necessary  and  then  adapt  the  supporting  architectures  (functional,  data- 
based,  organizational).  The  end  state  of  a  well  architected  framework  should  be  a 
coherent  set  of  processes  and  models  that  address  the  desired  capabilities. 

d.  Develop  Metrics  to  Fit  the  Framework 

With  a  well-defined  framework  the  last  step  should  be  to  select  metrics  for  the 
various  models.  Obvious  as  it  may  sound,  selecting  metrics  is  not  the  difficult  task;  it  is 
selecting  the  right  metrics.  The  entire  point  of  a  well-developed  data-based  architecture  is 
so  that  a  cogent  set  of  requirements  can  be  developed  to  guide  the  selection  of  metrics 
that  answer  the  right  questions.  While  this  is  perhaps  the  end  state  for  the  user,  it  should 
be  the  last  step  and  not  the  first  in  the  design. 

D.  FUTURE  WORK 

The  intent  of  this  section  is  to  outline  future  work  that  could  be  undertaken  as  a 
series  of  potential  follow  on  efforts  to  this  thesis.  Two  main  lines  of  research  are 
suggested.  The  first  is  the  work  that  would  likely  be  necessary  to  advance  the  sponsor’s 
objectives.  The  second  are  more  generalizable  lines  of  research  that,  if  undertaken,  would 
yield  great  benefit  to  a  wider  audience. 
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1.  Sponsor  Specific  Future  Work 

What  follows  is  a  suggested  way  forward  for  the  sponsor  to  address  the  stated 
problem.  Again  the  problem  as  stated  would  be  the  need  for  a  coherent  assessment 
framework.  The  sponsor  is  USSOUTHCOM,  but  as  discussed  in  Chapter  III,  any  other 
GPOI  implementer  and  certainly  the  DOS  GPOI  program  office  would  benefit  from  a 
sound  assessments  framework. 

Assuming  the  sponsor  holds  the  architecting  approach  and  SE  methods  to  be  valid 
means  to  address  the  problem,  two  principal  lines  of  effort  are  suggested  to  further 
advance  the  assessment  framework  as  put  forth  in  this  thesis:  (1)  problem  articulation  and 
assessments  framework  validation  (2)  metrics  selection.  These  lines  of  effort  can  be 
reasonably  furthered  either  by  the  sponsor  in  isolation  or  in  conjunction  with  follow  on 
research  efforts  at  NPS  or  any  other  qualified  consultant  for  that  matter. 

a.  Problem  Articulation  and  Assessments  Framework  Validation 

As  discussed  in  the  recommendations  sections,  the  sponsor  should  use  the 
proposed  framework  as  a  beta  or  a  point  of  departure  to  develop  a  robust  and  validated 
architecture.  This  can  perhaps  be  effectively  carried  out  by  another  consultant  (preferably 
someone  with  a  systems  thinking  orientation)  who  could  manage  the  refinement  of  the 
architecture  via  the  iterative  design  process  (see  Figure  3). 

The  idea  is  that  this  thesis’s  work  impacts  the  sponsor’s  thinking  to  the  degree 
that  the  requirement  set  can  made  more  precise  and  explicit;  the  improved  problem 
understanding  aides  the  developer  in  designing  a  better  supporting  architecture. 

b.  Model  Development  and  Metrics  Selection 

When  the  sponsor  is  satisfied  with  the  state  of  the  problem  articulation  and  the 
subsequent  architecture,  the  next  focus  area  should  be  building  the  necessary  models  to 
fill  out  the  assessments  framework.  Beta  models  are  offered  as  a  point  of  departure,  but 
will  likely  require  adaption  depending  on  the  direction  of  the  architecture. 
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2.  Future  Work  to  advance  Assessment  Theory 

What  follows  is  a  brief  discussion  of  future  research  topics  that  would  be 
generalizable  to  applications  outside  of  assessments  in  GPOI.  These  broader  topics  could 
be  explored  by  using  the  GPOI  as  a  use  case. 

a.  Exploring  the  Linkage  between  MOPs  and  MOEs  in  Assessment 
Development 

For  the  most  part  throughout  the  design  process  there  were  well  developed 
paradigms  or  methodologies  that  were  employed  directly  or  adapted  to  the  problem  at 
hand.  For  example,  during  the  problem  definition  phase  a  series  of  standard  SE  practices 
(stakeholder  analysis,  functional  analysis,  articulation  of  scope,  boundaries,  limitations, 
etc.)  were  used  to  give  the  problem  some  structure.  Also  during  the  architecting  phase 
there  were  several  frameworks  or  paradigms  that  were  used  as  a  starting  point  from 
which  to  build  an  assessments  framework  (this  thesis  used  an  IT  model,  see  Figure  9). 
However,  when  the  actual  models  were  being  developed,  a  glaring  hole  in  understanding 
became  obvious:  maintaining  traceability  from  measure  of  performance  (MOP)  to 
measure  of  effectiveness  (MOE). 

As  it  occurs  in  this  instance,  and  as  it  is  suspected  to  occur  in  many  others,  there 
is  a  fundamental  break  in  space  and  time  in  achieving  and  in  measuring  objectives  and 
outcomes.  In  the  GPOI  example  the  objectives  due  in  fact  link  to  the  notional  outcomes 
(see  Eigure  21).  What  is  unclear  is,  assuming  both  MOPs  and  MOEs  can  be  assessed: 
how  does  the  analyst  differentiate  between  causation  and  correlation  when  measuring 
effects  in  such  a  complex  system? 

This  is  a  problem  for  assessing  a  complex  system,  or  as  specifically  labeled  in  this 
thesis  an  unperceivable  system.  In  acquisition,  there  would  be  developmental  testing  for 
verification  (MOPs)  and  operational  testing  for  validation  (MOEs).  But  in  a  problem  set 
like  GPOI  where  the  assessment  is  complicated  by  being  part  of  such  an  intricate  system, 
it  would  seem  the  causation  piece  (that  is  maintaining  traceability  and  hence  validity  of 
the  assessment),  is  very  problematic. 
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b.  System  Dynamic  Modeling 

It  would  be  helpful  to  have  more  literature  on  the  subject  on  how  such  a 
complicated  system  could  be  modeled  and  better  understood.  At  one  point  in  this  thesis’s 
development,  the  idea  of  using  dynamic  modeling  to  evaluate  the  various  interactions 
was  considered.  There  is  readily  available  software  to  explore  the  system  along  these 
lines,  such  as  Stella  (IEEE  Systems  2014). 

A  follow  on  researcher  could  take  on  this  challenge  and  use  the  Global  Peace 
Operations  Initiative  as  use  case.  The  understanding  would  perhaps  yield  valuable  insight 
both  to  a  real  world  customer  as  well  as  potentially  shedding  light  on  new  ways  to  model 
and  understand  such  systems. 
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APPENDIX  A.  OVERVIEW  OF  GPOI  ASSESSMENT  PROJECT 

TEAM 


A.  RESEARCH  TEAM  ORGANIZATION 

The  research  team  is  an  interdisciplinary  team  composed  of  students  and  faculty 
from  multiple  departments.  The  vision  is  to  leverage  different  perspectives  from  separate 
academic  groups  in  the  hopes  of  being  better  equipped  to  prosecute  what  is  undoubtedly  a 
difficult  research  problem.  Students  from  both  the  Modeling,  Virtual  Environment  and 
Simulation  (MOVES)  Institute  and  the  Systems  Engineering  Department  are  participating 
at  both  the  master’s  and  doctoral  student  level. 

The  intent  is  for  the  doctoral  candidate  assigned  to  this  project  to  assist  in 
advising  the  master’s  thesis  while  at  the  same  building  understanding  of  the  problem  for 
follow  on  PhD  dissertation  work.  Thus,  the  master’s  thesis  (of  which  this  report  is  one)  is 
intended  to  be  part  of  the  foundational  work  of  both  future  dissertations  and  theses.  The 
longer  appointments  of  the  faculty  advisors  and  PhD  candidates  provide  continuity  to  the 
problem,  while  the  shorter  term  master’s  students  provide  a  continual  source  of  new 
perspectives. 

It  is  hoped  that  the  practical  advantage  of  this  organized  team  methodology  is  to 
allow  NPS  to  tackle  a  difficult  and  relevant  real  world  problem  in  a  concerted  manner 
that  otherwise  would  be  beyond  the  scope  of  any  one  single  thesis  or  dissertation. 

B.  SYNERGISTIC  RESEARCH  EFFORTS 

At  the  time  of  this  writing  another  master’s  thesis  on  the  subject  is  being 
conducted  in  parallel.  While  the  thesis  is  also  concerned  with  GPOI  assessments,  the 
research  focus  is  fundamentally  different  and  as  such  it  is  hope  that  the  two  efforts  will 
offer  unique  understanding.  Additionally,  the  beginnings  of  a  PhD  dissertation  are  also 
being  laid  out  on  the  topic  of  assessment  that  as  previously  stated  will  seek  to  leverage 
the  work  of  the  master’s  students. 


93 


c. 


RESEARCH  PLAN 


Figure  25  depicts  the  high  level  timeline  that  places  this  thesis  in  context  of  the 
broader  research  effort.  This  thesis  is  primarily  focused  on  architecting  an  assessment 
framework  with  associated  measurement  models.  Additionally,  another  SE  master’s 
student  is  analyzing  GPOI  from  a  systems  of  systems  view  while  a  MOVES  PhD  student 
is  concurrently  building  his  proposal. 


PHASE  I  GPOI  RESEARCH  PLAN 

JUN2013  JAN  2014  APR  2014 


NPS  Receives 
USSOUTHCOM 
Request  for  Analysis 


★ 


Stakeholder 
Discussions  (S) 
USSOUTHCOM 

A 


Intermediate  Product 
Review  (I  PR) 
w/USSOUTHCOM 


JUN  2014 

Phase  I  Product 
Delivery  to 
USSOUTHCOM 

/ 


Resident 

Systems 

Engineering 

Masters 

Students 


MOVES  PhD 
Student 


Background  Research 
&  Proposal  Dev. 


Background  Research 
&  Proposal  Dev 


Assessment 
y  Eramework  Thesis 
Development 

System  of  Systems 
£>)  Thesis 

Development 


Assessment 
Eramework  Thesis 
Refinement 

System  of  Systems 
Thesis  Refinement 


PhD  Dissertation  Background  Research  &  Proposal  Dev. 


PROJECT 

INCEPTION 


PHASE  I 
COMPLETE 


Figure  25.  GPOI  Phase  I  NPS  Research  Team  Schedule 


The  aforementioned  research  plan  references  phase  I  (from  which  this  thesis  was 
conducted  in),  however  a  follow  on  phase  is  planned.  Phase  II  will  have  a  similar 
structure  of  resident  master’s  students  conducting  focused  research  concurrently  with  the 
development  of  a  PhD  dissertation.  The  general  theme  of  phase  II  will  be  building  upon 
the  foundational  systems  engineering  efforts  of  phase  I  and  apply  modeling  and 
simulation  methods  to  further  enhance  understanding  and  yield  higher  fidelity  products. 

In  phase  II  the  sponsor  will  have  an  opportunity  to  refocus  the  requirement  set  in 
light  of  the  understanding  gleaned  from  phase  I.  The  researchers  will  be  benefited  by 
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groundwork  already  laid  as  well  as  a  less  compressed  scheduled  as  the  effort  will  not  be 
starting  from  a  standstill  and  will  have  a  longer  period  of  time  in  which  to  work.  A 
notional  phase  II  schedule  is  expressed  in  Figure  26. 


PHASE  I  &  II  GPOl  RESEARCH  PLAN 


JAN  2014 

APR  2014 

JUN  2014 

AUG  2014 

JAN  2015 

JUN  2015 

SEP  2015 

Project  Kickoff  @ 
USSOUTHCOM 

Phase  1  Intermediate 
Product  Review  (IPR) 

Phase  1  Product 
Delivery  to 

Phase  II 
focus/ redirect 

Phase  1  Intermediate 
Product  Review  (IPR) 

Phase  II  Thesis 
Delivery 

Phase  II  Dissertation 
Delivery 

Figure  26.  Notional  Phase  I  &  II  Research  Schedule 
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APPENDIX  B.  MODELING  PROCESS 


What  follows  are  the  key  elements  from  the  modeling  process  as  proposed  by 
Paul  West,  John  Kobza,  and  Simon  Goerger  (2011).  This  general  framework  was  applied 
throughout  the  thesis  as  a  general  methodology  for  thinking  about  and  constructing 
models  and  architectures.  Their  process  was  also  helpful  in  pointing  out  limitations  and 
potential  for  improvement  of  this  thesis’s  proposed  models  to  include  the  need  to  work 
iteratively  and  recursively  and  the  eventual  goal  of  verification  and  validation. 

A.  CREATING  A  CONCEPTUAL  MODEL 

The  conceptual  phase  is  concerned  with  the  enumeration  of  all  of  the  elements 
that  are  to  make  up  the  model  such  as  inputs,  outputs,  measures,  relationships,  processes. 
After  articulation,  diagramming  helps  to  further  flesh  out  the  necessary  elements  (West, 
Kobza  and  Goerger  2011). 

B.  CONSTRUCT  THE  MODEL 

The  construction  phase  is  carried  out  by  choosing  an  appropriate  modeling 
paradigm  (IDEFO,  FFBD,  SYSMF,  UMF,  etc.)  and  representing  the  elements  and 
relationships  from  the  conceptual  phase  (West,  Kobza  and  Goerger  2011).  This 
formalizes  the  models  and  adds  mathematical  coherency.  Clarity  of  presentation  was  the 
principle  factor  in  choosing  modeling  language. 

C.  EXERCISE  THE  MODEL 

Model  exercise  speaks  to  verification  and  validation  of  the  model  (West,  Kobza 
and  Goerger  2011).  This  would  require  a  degree  of  data  entry  and  ultimately  real  world 
operational  testing. 

D.  REVISE  THE  MODEL 

Revision  speaks  to  the  inherent  flexibility  of  a  model  (West,  Kobza  and  Goerger 
2011).  Once  created  engineers  and  end-users  should  be  able  to  update  the  models 
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throughout  the  lifecycle.  As  this  thesis  introduces  explicit  models  to  GPOI  assessment 

process,  revision  is  outside  of  the  scope  and  will  be  left  to  future  research  and  end-users. 

The  Modeling  Process  (according  to  West,  Kobza  and  Goerger  2011) 

Create  a  conceptual  model 

Identify  the  purpose  of  the  model 
Identify  the  input  variables 
Identify  the  output  measures 
Indentify  the  components  of  the  system 
Identify  controls 
Specify  assumptions 
Identify  relationships  and  interactions 
Draw  a  diagram  of  the  system 
Create  a  flow  chart  of  the  system 

Construct  the  model 

Choose  a  model  type 
Represent  relationships 

Exercise  the  model 
Verify 
Validate 
Accredit 

Revise  the  model  (model-test-model) 
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APPENDIX  C  CLASSIC  SE  V-MODEL 


Figure  27  is  a  representation  of  the  well-known  Systems  Engineering  V-Model 
(Clark  2009).  It  explains  both  “top  down”  and  “reverse”  engineering  processes  for 
systems.  The  illustrative  point,  as  it  pertains  to  the  Global  Peace  Operations  Initiative,  is 
that  if  the  assessments  framework  is  operating  only  at  the  verification  level,  both  in  its 
design  and  implementation,  then  it  has  effectively  cut  out  the  customer  requirements  as 
well  as  ultimate  end  state  of  the  system.  Hence  the  goal  is  to  architect  a  framework  that  is 
end  to  end  and  expresses  the  arc  of  desired  outcomes  to  measured  outcomes. 


Figure  27.  Classic  Systems  Engineering  V-Model  (from  Clark  2009) 


99 


THIS  PAGE  INTENTIONALLY  LEET  BLANK 


100 


APPENDIX  D.  GPOI  PHASE  I  OBJECTIVES 


PHASE  I  GPOI  OBJECTIVES 

1.) 

Train  and,  as  appropriate,  equip  at  least  75,000  peacekeepers  by  2010,  with  an 

emphasis  on  Africa; 

2.) 

Enhance  regional  capacities  and  support  institution  building; 

3.) 

Support  the  G8  Africa  Clearinghouse  and  establish  a  G8+  Global  Peace 
Support  Operations  Capacity  Building  Clearinghouse; 

4.) 

Support  the  development  of  a  G8  Transportation  and  Eogistics  Support 

Arrangement; 

5.) 

Develop  a  cached/deployment  equipment  program; 

6.) 

Support  Italy’s  Center  of  Excellence  for  Stability  Police  Units  (COESPU);  and 

7.) 

Conduct  self-sufficiency  and  sustainment  efforts  in  support  of  all  activities 

listed  above 

Table  2.  Phase  1  GPOI  Objectives  (after  U.S  Department  of  State  n.d.) 
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APPENDIX  E.  HISTORICAL  FUNDING  LEVELS  FOR  THE  GPOI 


GPOI  HISTORICAL  FUNDING  LEVELS  BY  EISCAL  YEAR 

EY05 

$96,666,000 

EY06 

$100,384,000 

EY07 

$81,000,000 

EY08 

$96,442,437 

EY09 

$105,900,000 

EYIO 

$96,900,000 

EYII 

$98,800,000 

EY12 

$91,850,000 

EY13 

$75,000,000 

Table  3.  GPOI  Historical  Funding  Levels  Represented  as  Congressional 

Appropriations  (after  Bureau  of  Political-Military  Affairs  and  Office 
of  Plans,  Policy  and  Analysis  2013) 

The  tables  are  from  the  State  Department’s  GPOI  Implementation  Guide  (Bureau 
of  Political-Military  Affairs  and  Office  of  Plans,  Policy  and  Analysis  2013) 
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APPENDIX  F.  FIRST  ORDER  STAKEHOLDER  ANALYSIS 


Table  4.  First  Order  Stakeholders  for  GPOI  Assessment 
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APPENDIX  G 


GPOI  EXPANDED  FUNCTIONAL  HIERARCHY 


Figure  28.  Expanded  GPOI  Functional  Hierarchy 
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APPENDIX  H.  GPOI  DATA  FLOW  DIAGRAM 


Figure  29,  a  working  data  flow  diagram  (DFD)  for  GPOI  assessments,  is  an 
intermediate  abstraction  used  by  the  modeler  to  begin  thinking  about  the  nature  of  the 
connections  of  the  stated  system  functions  (Buede  2000).  The  DFD  was  used  to  facilitate 
the  translation  of  the  functional  hierarchy  (see  Figure  5)  into  a  defined  functional  model 
(see  Figure  6)  (Trainor  and  Parnell  2011). 
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APPENDIX  I.  IDEFO  PRIMER 


IDEFO  was  used  for  functional  modeling  in  this  these  was  primarily  used  for 
clarity  of  presentation.  On  the  clarity  front  IDEFO  focuses  on  the  relationships  of 
functions  or  activities  in  a  way  that  represents  a  larger  process.  As  this  thesis  is  concerned 
with  understanding  how  the  assessment  function  operates  within  the  greater 
organizational  process  of  the  GPOI,  IDEFO  is  a  natural  choice. 

IDEFO  links  functions  or  activities  together  by  defining  the  nature  of  their 
connections.  The  connections  of  each  function  are  categorized  in  four  bins:  inputs, 
outputs,  controls,  and  mechanisms  (see  Figure  30).  The  output  is  what  the  function  is 
required  to  achieve  with  respect  to  the  rest  of  the  system.  The  input  is  what  the  specific 
function  requires  to  execute  its  activity.  The  mechanism  is  the  entity  (e.g.,  person, 
organization,  and  equipage)  that  will  carry  out  the  function.  Finally,  the  control  are  the 
conditions  or  set  of  conditions  required  to  achieve  the  desired  output  (e.g.,  policies, 
approval  criteria)  (Software  Engineering  Standards  Committee  of  the  IEEE  Computer 
Society  1998). 


CONTROLS 


-INPUTS - ► 


Function/ 

Activity 


-OUTPUT(S)- 


MECHANISM(S) 


Figure  30.  Basic  IDEFO  Structure 
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APPENDIX  J.  FUNCTIONAL  ANALYSIS  DISCUSSION 


The  functional  analysis  (see  Figure  9)  produced  a  series  of  system  level 
requirement  and  insights  that  will  be  helpful  in  architecting  a  suitable  assessments 
framework  (as  developed  in  Chapter  IV).  Again,  these  insights  are  caveated  as  being 
valid  to  the  degree  with  the  functional  analysis  is  sound. 

A.  DESCRIPTIVE  UNDERSTANDING 

As  mentioned  in  Chapter  I,  every  product  can  be  binned  into  two  main  product 
families,  prescriptive  and  descriptive.  Prescriptive  products  propose  modifications  to  the 
system.  Descriptive  products  articulate  the  “as  is”  or  “should  be”  condition.  The  IDEFO 
functional  model  in  Figure  6  is  a  descriptive  product  as  it  was  built  based  on  the 
understanding  of  GPOI  operations  based  on  program  policy. 

If  the  sponsor  believes  the  model  to  be  deficient,  then  either  the  model  itself  is  not 
expressed  properly  or  the  real  world  practice  is  perhaps  lacking  (or  both).  In  the  former 
case,  the  solution  is  to  of  course  amend  the  model  and  thus  a  more  useful  abstraction  of 
organizational  process  is  achieved.  If  in  the  latter  case,  that  is  practice  does  not  conform 
to  policy,  then  the  sponsor  should  evaluate  policy  and/or  the  organizational  processes 
being  implemented. 

B.  MEASUREMENT  BASIS 

The  functional  analysis  offers  very  intuitive  logic  chains.  The  first  is  that  policy  is 
the  starting  point  from  which  objectives  are  built,  from  which  activities  should  be  based 
on.  This  offers  clear  traceability  and  basis  of  measurement.  Most  importantly  it  is  offered 
that  the  outcomes  of  the  events  activities  should  be  measured  against  the  stated 
objectives.  Even  the  objectives  themselves  can  be  measured  based  on  the  articulated 
overarching  policy. 
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c. 


ROLE  OF  ASSESSMENTS 


Within  the  system  the  assessment  functions  serves  as  feedback  mechanisms.  The 
use  of  the  information  is  different  depending  on  which  is  activity  is  consuming  the 
assessment.  For  example  perhaps  the  “Obtain  Funding”  function  would  use  the 
assessments  as  justification  before  Congress.  The  implementer  set  could  use  the 
information  as  a  performance  assessment  as  to  how  impactful  they  are  actually  being. 
And  for  the  strategic  function,  the  assessment  exists  for  strategic  redirection  and  program 
modification.  While  different  nodes  in  the  system  will  use  the  assessment  differently  it  is 
offered  that  they  all  require  the  fundamentally  same  data  set. 

The  implication  is  twofold.  The  first  is  transmission.  If  the  assessed  information  is 
not  being  properly  promulgated  for  higher  system  use  then  it  does  not  matter  how  good 
the  assessment  is.  The  second  is  the  type  of  information.  It  does  not  suffice,  of  course,  to 
just  do  an  assessment.  As  previously  mentioned  what  is  required  is  the  right  information 
that  shows  traceability  and  hence  causality. 

If  the  wrong  information  is  being  collected  or  perhaps  the  right  information  is  not 
properly  disseminated  through  the  system  then  the  assessment  function  essentially 
becomes  what  is  referred  to  as  a  “self-licking  ice  cream  cone.”  In  other  words  if  the 
framework  is  not  set  up  to  extract  specific  forms  of  information  and  then  communicate 
them  to  the  other  nodes  then  the  assessment  activity  will  in  effect  be  an  untethered 
function  that  is  not  serving  the  broader  system  properly. 

D.  IMPORTANCE  OF  MEASUREMENT  VARIETY 

The  model  as  proposed  shows  the  assessment  function  measuring  various  parts  of 
the  system.  The  implication  is  that  a  variety  of  information  will  likely  be  required  to 
generate  sound  assessments.  This  is  of  course  not  a  profound  statement,  but  is  none  the 
less  made  to  emphasize  the  importance  of  extracting  the  right  information  from  the  right 
parts  of  the  system. 

There  need  not  necessarily  be  a  large  quantity  of  information  gathered,  just  the 
right  information.  To  paraphrase  Robert  Michael’s  analysis  of  assessments  conducted  in 

the  Iraq  and  Afghanistan  conflicts,  excessive  and  extraneous  information  is  detrimental. 
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There  is  perhaps  a  temptation  to  measure  what  can  be  readily  measured,  but  this  should 
be  avoided  in  favor  of  attempting  to  measure  what  is  required.  (Michael  2010) 

E.  CONTROLS  AND  MECHANISMS 

One  of  the  benefits  of  modeling  a  system  in  IDEFO  is  that  forces  the  SE  to 
consider  carefully  the  controls  and  mechanisms  required  of  each  function;  the  notion  that 
functions  or  activities  by  definition  have  controls  and  mechanism  is  a  defensible  enough. 
That  is  every  function  must  be  executed  by  some  entity  (mechanism)  and  it  usually  does 
so  under  some  level  of  guidance  or  constraint  (controls).  Thus  is  it  offered  here  that  for 
the  assessment  function  within  GPOI  the  mechanism  would  be  the  assessment  personnel 
themselves  (which  are  of  course  drawn  across  multiple  levels  and  agencies).  The  control 
would  be  the  assessment  framework.  The  assessment  framework  would  be  the  policies 
and  process  governing  assessments  as  well  as  the  assessment  models  themselves. 
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APPENDIX  K.  GPOI  ASSESSMENTS  OPERATIONAL  CONCEPT 


Figure  31  is  an  operational  concept  graphic  or  OV-1  of  Assessments  in  the  Global 
Peace  Operations  imitative.  An  OV-1  is  a  defined  view  in  the  Department  of  Defense 
Architectural  Framework  (DODAF).  The  OV-1  is  meant  to  convey  the  big  picture  of  a 
systems  actualization  to  a  wide  audience.  There  are  of  course  many  more  elements  and 
relationships  that  could  be  included  and  defined,  but  the  intent  is  to  convey  a  high  level 
view.  For  the  system’s  architect  this  is  a  useful  intermediary  view  to  begin  thinking  about 
the  system  (Dam  2006). 
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Figure  3 1 .  GPOI  Assessments  Operational  Concept  (OV-1) 
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APPENDIX  L.  WORKING  METRICS  REQUIREMENTS 

DOCUMENT 


1 .  Define  Metrics  for  GPOI  Models 

1.1.  Define  Metrics  for  Plans  and  Policy  Assessment — This  requirement  is  achieved 
by  having  a  defined  capability  to  assess  whether  GPOI  is  articulating  the  correct 
high  level  policy  to  govern  the  program  as  well  as  whether  or  not  the  stated 
processes  and  procedures  align  with  the  policy  as  articulated. 

1.1.1.  Define  Metrics  for  Assessing  for  GPOI  Policy  and  Objectives - The 

question  to  answer:  is  the  high  level  GPOI  policy  correctly  articulated  in 
view  of  higher  level  national  policy  directives  and  international  agreements? 

1.1.2.  Define  Metrics  for  Assessing  GPOI  Processes  &  Procedures — The 
question  to  answer:  (I)  Are  the  stated  processes  and  procedures  in 
accordance  with  GPOI  policy?  and  (2)  are  the  missing  processes  and 
procedures  that  need  to  be  made  explicit? 

1.2.  Define  Metrics  for  Input  Assessment — This  requirement  is  achieved  by  the 
capability  of  quantitatively  assessing  the  input  or  the  cost  of  the  GPOI  program 

1. 2. 1.  Define  Metrics  for  Measuring  Monetary  Input — the  question  to  answer  is: 
How  much  does  the  event,  or  element,  of  the  GPOI  cost? 

1.2.2.  Define  Metrics  for  Measuring  Organizational  Input  -The  question  to 
answer  is:  how  much  organizational  bandwidth  was  consumed  by 
conducting  the  event?  Recommended  to  either  monetize  this  value  or 
represent  as  an  opportunity  cost 

1.3.  Define  Metric  for  Process  Assessment — This  requirement  is  achieved  by  the 
capability  of  assessing  how  a  particular  event  conformed  to  stated  policy 

1.3.1.  Define  Metrics  for  measuring  adherence  to  stated  processes — The 
question  to  answer  is:  was  the  event  conducted  in  accordance  with  GPOI 
policies  and  procedures? 

1.4.  Define  Metrics  for  Output  Assessment — This  requirement  is  achieved  by  the 
capability  of  assessing  how  a  particular  event,  or  series  of  event,  achieved  stated 
objectives: 

1.4.1.  Define  Metrics  for  Assessing  Achievement  of  Objective  #1 — The  question 
to  answer  is  how  did  the  event  contribute  to  the  partner  country  achieving 
FTC? 

1.4.2.  Define  Metrics  for  Assessing  Achievement  of  Objective  #2 — The  question 
to  answer  is  how  did  the  event  contribute  to  training  peacekeepers? 

1.4.3.  Define  Metrics  for  Assessing  Achievement  of  Objective  #3 — The  question 
to  answer  is  how  did  the  event  contribute  to  providing  support  for  deploying 
units? 

1.4.4.  Define  Metrics  for  Assessing  Achievement  of  Objective  #4 — The  question 
to  answer  is  how  did  the  event  enhance  the  peace  operations  capacity  of 
regional/sub  regional  organizations  and  institutions? 
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1.4.5.  Define  Metrics  for  Assessing  Achievement  of  Objective  #5  -The  question 
to  answer  is  how  did  the  event  establish  or  strengthen  the  institutional 
infrastructure  and  doctrinal  framework  to  train,  equip,  and  deploy  FPUs? 

1.4.6.  Define  Metrics  for  Assessing  Achievement  of  Objective  #6 — The  question 
to  answer  is  how  did  the  event  support  the  continuation  and  enhancement  of 
multilateral  approaches/partnerships  to  coordinate  peace  operations  capacity 
building  efforts? 

1.5.  Define  Metrics  for  Outcome  Assessment — This  requirement  is  achieved  by  the 
capability  of  assessing  the  level  of  outcome  attainment. 

1.5.1.  Build  Metrics  that  link  back  to  Stated  Objectives 

1.5. 1.1.  Define  Metrics  for  Assessment  of  Notional  Outcome  #1 — The 
question  answer  is  how  has  regional  security  been  enhanced? 

1.5. 1.2.  Define  Metrics  for  Assessment  of  Notional  Outcome  #2 — The 
question  to  ask  is:  how  successful  were  the  GPOI  trained  units  on  of 
UN  peacekeeping  missions? 
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APPENDIX  M.  NOTIONAL  ORGANIZATIONAL  RESOURCE 

MANAGEMENT  MODEL 


Table  5.  Notional  Organizational  Resource  Management  Plan 
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